Reference Manual

Introduction

The NMI framework software works on top of an existing Condor pool. You must identify hosts to run the following NMI facility services:

Condor Central Manager host
One or more combined Submit/Archive/Web Server hosts (currently these must be co-located)
One DB host

1 to n Execute hosts

Any/all of these services may be co-located on the same host if desired. It’s recommended to have at least 1 execute host to start.

Metronome Installation

Preparing System Environment

If you are using Linux for your submit node, you must install the mysql-dev and Perl DBI modules.

perl -MCPAN -e "install DBI"
perl -MCPAN -e "install DBD::mysql"

Preparing the Database

Install the version of MySQL as listed in the release notes. You will need to create a database with the same name as defined DB_NAME (the default name is nmi_history) and install the default schema.

mysqladmin create nmi_history
mysql nmi_history < nmi-X.Y.Z/framework/database/schema.mysql

Now as a privileged user, create the following accounts. The first account shown below is the DB_WRITER_USER, and it needs to be able to insert and update records in the database. The second account, DB_READER_USER, is use by the web interface and needs only read access to the database, except to update the “notes” field (which you can turn off), and to write to the sessions table (which, at least for now, you can’t).

Replace ‘@%.example.com@’ with the appropriate domain or host. Only hosts you specify for the DB_WRITER_USER will be able to use the command-line tools, and only hosts you specify for the DB_READER_USER will be able to run the web interface.

NOTE: Be sure to execute FLUSH PRIVILEGES; to make sure these accounts are add appropriately. You may also need to create an additional ‘localhost’ record for each account if the database is running on the same host as the submit node.

# DB_WRITER ACCOUNT
GRANT SELECT,INSERT,UPDATE,DELETE ON nmi_history.* \
TO 'DB_WRITER_USER'@'.example.com' IDENTIFIED BY 'DB_WRITER_PASS'; 

# DB_READER ACCOUNT
GRANT SELECT,CREATE TEMPORARY TABLES ON nmi_history.* \
TO 'DB_READER_USER'@'.example.com' IDENTIFIED BY 'DB_READER_PASS'; 
GRANT UPDATE (notes) ON nmi_history.Run \
TO 'DB_READER_USER'@'.example.com' IDENTIFIED BY 'DB_READER_PASS';
GRANT SELECT,INSERT,UPDATE,DELETE ON nmi_history.sessions \
TO 'DB_READER_USER'@'.example.com' IDENTIFIED BY 'DB_READER_PASS';

Installing NMI Framework

Install the NMI software under your chosen prefix:

perl Makefile.PL prefix=<prefix>
make install

If you anticipate installing multiple versions of the framework, you may wish to set the prefix to a location such as /nmi-x.y.z, then create symbolic links to the installation directories:

mkdir <prefix>/nmi
cd <prefix>/nmi
ln -s <prefix>/nmi-X.Y.Z/share
ln -s <prefix>/nmi-X.Y.Z/bin
ln -s <prefix>/nmi-X.Y.Z/lib

Framework Configuration

Copy nmi-X.Y.Z/framework/nmi.conf.sample to prefix/etc/nmi.conf and edit as required. Please make sure that all non-trivial configuration parameters are customized for your local site (see Site Configuration Parameters for more information).

mkdir <prefix>/etc
cp nmi-X.Y.Z/framework/nmi.conf.sample <prefix>/etc/nmi.conf
edit <prefix>/etc/nmi.conf

If you intend to install future framework versions, you may want to place your nmi.conf file in a general location such as /nmi/etc and create the symlink from prefix/etc instead:

mkdir /nmi/etc/@
cp <prefix>/etc/nmi.conf /nmi/etc/
cd <prefix>/etc
ln -s /nmi/etc/nmi.conf nmi.conf

Hawkeye modules setup

The framework relies on Condor Hawkeye technology to ensure that jobs get matched to machines with the right platform. Put the following lines in your Condor config file on ALL of your EXECUTE hosts:

# EDIT_ME: In the next line, is a directory
# path where you keep your Hawkeye modules, if any. For example, it
# could be /home/condor/hawkeye_modules.
MODULES =
STARTD_CRON_NAME = NMIPOOL
# Uncomment the following line if NMIPOOL_JOBS has not been defined yet.
# NMIPOOL_JOBS =
# JOB: Report the list of software installed on the system.
NMIPOOL_JOBS = $(NMIPOOL_JOBS) prereq:has_:$(MODULES)/prereq:10m:kill
# EDIT_ME: In the next line, is the path to the
# directory containing individual prereq installations. For example,
# it could be /prereq.
NMIPOOL_PREREQ_PREREQDIR =
# JOB: Report the nmi_platform.
NMIPOOL_JOBS = $(NMIPOOL_JOBS) nmi_platform::$(MODULES)/nmi_platform:720m:kill

Now take the contents of the framework ‘hawkeye_modules’ directory that comes with this distribution and, on ALL of your EXECUTE hosts, copy the files to .

Check that the module returns sensible values when run directly on your build machines. For example:

./nmi_platform
nmi_platform = "ppc_macos_10.3"
—

Now restart Condor on the execute hosts and verify that they report their NMI platform correctly to Condor Collector. You should be able to see something like:

condor_status -l | grep nmi_platform | head -5
nmi_platform = "ppc_aix_5.2"
nmi_platform = "hppa_hpux_B.10.20"
nmi_platform = "irix_6.5"
nmi_platform = "alpha_rh_7.2"
nmi_platform = "ia64_rhas_4"

Similarly, to test the prereq module, install some prereqs in your prereqs_location – for example, let’s say you installed python-2.2.3 from source using —prefix=/prereq/python-2.2.3 option to configure. Then you should be able to see something like:

./prereq | grep python-2.2.3
python_2_2_3 = "/prereq/python-2.2.3"

Similarly, if in the command below you replace by the actual hostname for that machine, you should be able to see something like:

condor_status -l | grep python
has_python_2_2_3 = "/prereq/python-2.2.3"

Advanced Hawkeye Usage

You can use Hawkeye to help match jobs to machines on the basis of attributes other than platform. We wrote a Hawkeye module, publish_dir.pl (see below), to simplify this task. It reads files from a specified directory, ignoring lines beginning with ‘#’ (or whitespace followed by ‘#’) and lines which don’t match the ‘attribute = value’ form. For those lines, it makes ‘attribute’, with the value ‘value’, available for matching jobs to the machine.

For example,

# ETICS includes compiler information in the platform string.
etics_platform = Darwin821_powerpc_gcc400

if you set the PLATFORM_TYPES configuration file variable to ‘etics’, users could specify ‘Darwin821_powerpc_gcc400’ as the platform for their jobs.

How to create these files is left for the particular situation of the reader, although it will be convient in many cases to use Hawkeye. See the Condor manual.

Running `publish_dir.pl`

For older versions of Condor (pre-6.9.x), the following should work:

NMIPOOL_JOBS = $(NMIPOOL_JOBS) publish_dir::$(MODULES)/publish_dir.pl:1h:kill
NMIPOOL_PUBLISH_DIR_PATH = /prereq/.hawkeye

This Condor configuration snippet applies to later versions of Condor:

# Report attribute = name pairs from the named directory.
NMIPOOL_JOBLIST = $(NMIPOOL_JOBLIST) publish_dir
NMIPOOL_PUBLISH_DIR_PREFIX =
NMIPOOL_PUBLISH_DIR_EXECUTABLE = $(MODULES)/publish_dir.pl
NMIPOOL_PUBLISH_DIR_PATH = /prereq/.hawkeye
NMIPOOL_PUBLISH_DIR_PERIOD = 1h
NMIPOOL_PUBLISH_DIR_KILL = True

In both cases, publish_dir.pl update the attribute-value pairs available for matching, as specified by the files in /prereq/.hawkeye, once an hour.

Configuring a Metronome Pool for Parallel Testing

Overview

This feature of Metronome builds on the Condor Parallel Universe and provides for running jobs on multiple machines simultaneously. Condor's Chirp mechanism makes communication between the machines possible.

Version Requirements

Condor

7.1.0 - latest stable series release, adds "parallel job exit policies":node/1296
7.0.1 - latest developer series release, adds "parallel job exit policies":node/1296
6.9.5 - broken for // jobs, do not use!
6.9.0 - adds global scratchpad feature
6.7.20 - fixes Node 0 killing other nodes bug
6.7.19 - fixes Scheduler attribute
6.7.18 - first Condor release with Parallel universe support for DAGMan

Metronome

2.5.0 - latest developer series release, recommended.
2.4.4 - latest stable series release
2.4.3 - nmi_putattr and nmi_getattr bugs, do not use!
2.2.3 - uses global scratchpad feature
2.x.x - includes native support for Parallel universe jobs

Pool Setup

Each Metronome pool where you would like to run parallel jobs requires 1 DedicatedScheduler machine. The DedicatedScheduler is usually a submit node, as the Parallel Universe is based on the condor_schedd daemon. Each execute machine must then know about the submitter. There are several ways to configure your pool to run parallel jobs.

Add 1 P-slot to each machine in your pool. For this example, assume that you already have 2 multi-use slots set up on each execute machine and you are adding slot 3 to run only parallel jobs.

Replace the LOCAL_CONFIG_FILE line in your condor_config file:

    #LOCAL_CONFIG_FILE      = $(LOCAL_DIR)/$(HOSTNAME).local
    PARALLEL = $(LOCAL_DIR)/condor_config.parallel
    LOCAL = $(LOCAL_DIR)/$(HOSTNAME).local

    REQUIRE_LOCAL_CONFIG_FILE = True
    LOCAL_CONFIG_FILE = $(PARALLEL), $(LOCAL)

Create a config file called condor_config.parallel and put it on all machines where you would like to run parallel jobs. Add the following lines, replacing submit-host.your.org with the name of the machine you will be submitting from.

######################################################################
# MPI Settings
######################################################################

##  If you want to "lie" to Condor about how many CPUs your machine
##  has, you can use this setting to override Condor's automatic
##  computation.  If you modify this, you must restart the startd for
##  the change to take effect (a simple condor_reconfig will not do).
##  Please read the section on "condor_startd Configuration File
##  Macros" in the Condor Administrators Manual for a further
##  discussion of this setting.  Its use is not recommended.  This
##  must be an integer ("N" isn't a valid setting, that's just used to
##  represent the default).
NUM_CPUS = 3

##  The number of evenly-divided virtual machines you want Condor to
##  report to your pool (if less than the total number of CPUs).  This
##  setting is only considered if the "type" settings described above
##  are not in use.  By default, all CPUs are reported.  This setting
##  must be an integer ("N" isn't a valid setting, that's just used to
##  represent the default).
NUM_VIRTUAL_MACHINES = $(NUM_CPUS)

##  What is the name of the dedicated scheduler for this resource?
DedicatedScheduler = "DedicatedScheduler@submit-host.your.org"

##  Path to the special version of rsh that's required to spawn MPI
##  jobs under Condor.  WARNING: This is not a replacement for rsh,
##  and does NOT work for interactive use.  Do not use it directly!
MPI_CONDOR_RSH_PATH = $(LIBEXEC)

##  This setting puts the DedicatedScheduler attribute, defined above,     
##  into your machine's classad.  This way, the dedicated scheduler
##  (and you) can identify which machines are configured as dedicated
##  resources.
SLOT1_STARTD_EXPRS = $(STARTD_EXPRS)
SLOT2_STARTD_EXPRS = $(STARTD_EXPRS)
SLOT3_STARTD_EXPRS = DedicatedScheduler

## required so the start expr won't eval to false and prevent jobs from ever running.
IsOwner = False

## Be cautious that you don't override this START expression in other condor_config.* files.
START = ( (VirtualMachineID == 1) || \
          (VirtualMachineID == 2) || \
          (VirtualMachineID == 3) && $(SLOT3_TYPE) )


## slot3 runs parallel / MPI jobs
SLOT3_TYPE = (Scheduler =?= $(DedicatedScheduler))

Allow all of the slots to run parallel jobs as well as other jobs.
TBD.

Configuring jobs to require certain hosts, or visa-versa

Matchmaking is bilateral — that is, both jobs and hosts can express their own requirements of one another, and can advertise their own attributes for one another’s reference.

A platform_job’s Requirements expression specifies its requirements of a host to run on. In Metronome, you can easily add a new constraint to the platform_job’s Requirements expression via the append_requirements command in the run specification file, like so:

append_requirements = (host_attribute =?= "bar")

This tells Condor to make sure that the job only runs on a host whose host_attribute equals “bar”.

Likewise, a host’s START expression specifies its requirements of a job. To add a new constraint to the host’s START expression, you should edit the host’s condor_config (or condor_config.local) file, like so:

START = ( $(START) && job_attribute =?= "foo" )

This tells Condor to make sure that the host only receives jobs whose job_attribute equals “foo”.

To advertise a new job attribute (so you can reference it in a host’s START expression), just add it via the ++ command in the run specification file, like so:

++job_attribute = "foo"

To advertise a new host atttribute (so you can reference it in a job’s Requirements expression), just add it to the host’s condor_config (or condor_config.local) file. like so:

host_attribute = "bar"
STARTD_EXPRS = $(STARTD_EXPRS) host_attribute

Additional Notes

What is `=?=` and How Is It Different From `==`?

The short answer: Use =?=, not ==.

The long answer: In the boolean logic of Condor classads, expressions can evaluate to True, False, or Undefined. If an expression references an attribute which isn’t defined, the value of that expression becomes Undefined. Therefore, if you say:

START = (job_color == "Red")

...and job_color is not defined in the job ad, then START will evaluate to Undefined rather than false.

To avoid this, Condor classads provides a =?= variant of the equality operator which will evaluate to False if one half is Undefined, rather than evaluating to Undefined. So if you say:

START = (job_color =?= "Red")

...and job_color is not defined in the job ad, then START will evaluate to False rather than Undefined.

Likewise for =!= and !=.

Debugging

Although it’s not needed for correctness, for debugging you might also want to add the following to the host’s condor config_config:

STARTD_JOB_ATTRS = $(STARTD_JOB_ATTRS) job_attribute

This will tell the host to publish the job_owner attribute of any currently running job in its own host classad, so you can see it. This can make it easier to confirm that the job that is currently running has the attribute you expect, without having to look up the jobid and examine its classad separately using condor_q -l. The only thing to be careful about is any attribute names which are present in both the host and job classads, because only one can be published in the machine classad. This is one good reason to name job attributes and host attributes with job_ or host_ prefixes.

Cross-Site Job Migration

This feature in Metronome allows build and test submissions to automatically migrate to different pools based on resource availability. If a local user submits a job that requests a particular platform that does not exist in the local pool, it can automatically be routed to a different site that does have a computing resource with that platform.

The following information and pages are a guide for system administrators and pool managers to allow their local Metronome installation to establish routes with others sites.

Overview

There are two major components of the job migration feature:

Broadcast local resource information to other sites

Creating a routing table for local jobs to remote sites

This is preformed by the nmi_resource_advertiser tool. It transmits information about your local Metronome resources to remote collectors, such as the number of available machines in the pool and which type of platforms are available. Other sites that you have a pair-wise agreement with will in turn broadcast their pool information back to your local collector.

With this information, the resource advertiser can construct a table of routes to the remote sites where local jobs can execute on in order to get the resources they need.

To enable this feature, there are several steps and configuration changes that you will need to make to your local Metronome installation. Each pool that you wish to route jobs to will also need to make the same changes.

Limitations

Routes are currently bi-directional. There is no way to have jobs migrate from Site A to Site B but not have jobs migrate from Site B to Site A.

There is no way to prevent your job from executing on remote sites in the current version. All jobs are eligible for migration after remaining idle for a certain period of time if there are resources that match their job.

Jobs are only matched based on the platform that they requested. The system is unable to make matches based on prerequisite software requirements.

There is currently no way to specify which platforms are allowed to be shared with remote sites.

NMI Configuration

NMI Configuration File

There are two parameters that need to be added to the end of your nmi.conf file on each of your pool’s submit nodes:

## ---------------------------------------------
## Job Routing
## ---------------------------------------------
ROUTING_TABLE = /path/to/condor/local/dir/condor_config.routing_table
REMOTE_SITES =  /path/to/nmi/etc/remote_sites

The routing table is populated by the resource advertiser with information about how to migrate jobs to remote sites (see this page for more information). It is important that the ROUTING_TABLE entry be writable by the nmi_resource_advertiser script and readable by the Condor daemons. The remote_sites entry should be placed in the same directory as the submit node’s nmi.conf file.

Remote Sites List

The REMOTE_SITES file is a list of hostnames that your local pool is allowed to route jobs to. Each line in the file should contain a hostname of a remote submit node and optionally the hostname of the collector to broadcast the resource information to for the submit node. If no collector host is provided, the resource advertiser will send all information to collector listening on the submit node at the standard port.

Please note that the entries in this file must match exactly with the hostname on each machine. The resource advertiser uses this list to look for remote resource information in the local collector and also to send the local resource information to each of these sites.

Sample list:

# Comments are allowed
#     
remote.site1.com
remote.site2.com   collector.site2.com
remote.site3.com   collector.site3.com:1234

Submit Node Configuration

Cron Job

The nmi_resource_advertiser program is used to broadcast information about the local NMI resources to sites listed in the REMOTE_SITES file and write the routes to remote sites to the ROUTING_TABLE used by the job router.

You will need to add the following command to the crontab of the user that Condor runs as (on most systems this is usually condor or daemon). The command will execute every five minutes. This is just a default interval that should suit most configurations. It should be less than the ClassAd lifetime of information stored by the Collector (CLASSAD_LIFETIME).

*/5 * * * * /path/to/bin/nmi_resource_advertiser --broadcast --routing-table --nmiconf=/path/to/nmi/etc/nmi.conf

Firewall Configuration

If all of your submit and worker nodes are located either on the public network or behind a firewall, there are no configuration changes needed for Metronome. If, however, worker nodes are installed at remote sites that have firewalls, then you will need to configure both Condor and the firewalls to allow the appropriate network traffic. Note that the following changes must be made in your site’s Condor configuration file and not nmi.conf

Please refer to the Networking Section of the Condor Manual for the most up-to-date information about what ports are needed by Condor.

Setting Condor’s Listening Port Range

Condor can be configured to open sockets within a range of values. Please refer to the information below on the number of ports that need to be opened for your installation. The following example will cause Condor to only open ports

LOWPORT = 20000
HIGHPORT = 25000

The range of ports assigned may be restricted based on incoming (listening) and outgoing (connect) ports with the configuration variables IN_HIGHPORT, IN_LOWPORT, OUT_HIGHPORT, and OUT_LOWPORT.

Local Inclusion of Remote Resources

The most important port to open is 9618; this is used as the listening port of the Condor Collector for receiving resource information from worker nodes.

There are several configuration possibilities for included remote resources. For example, you may just allow users to add their resources to your Metronome site’s pool, or you may allow them to submit jobs from the outside into your pool.

The worker nodes that are outside of the

Remote Worker Nodes Connecting to Local Collector

Description needed here…

Number of ports needed:

5 + (5 * number of virtual machines advertised by that machine)

Remote Submit Nodes Connecting to Local Resources

Description needed here…

Number of ports needed:

5 + (5 * MAX_JOBS_RUNNING)

Local Central Manager Connecting to Remote Resources

Description needed here…

Number of ports needed:

5 + NEGOTIATOR_SOCKET_CACHE_SIZE

Job Migration

The most important port to open is 9618; this is used as the listening port of the Condor Collector for receiving resource information from worker nodes.

Metronome Configuration Parameters

The Metronome configuration on each submit/archive node is determined by the settings in the nmi.conf configuration file. This page is a list of all the available parameters.

Any text after a hash character (#) is considered a comment. The syntax for each line in the configuration file is:

# Comments are ignored
 =    # inline comment

NOTE
Starting in Metronome 2.2.0, the following parameters are deprecated. The default configuration file now uses UPPERCASE naming, although the lower case versions will still be supported. Use the appropriate substitution when updating your configuration file:

database -> DB_NAME
mysqlhost -> DB_HOST
mysqlport -> DB_PORT
username -> DB_USER
password -> DB_PASS
nmiprefix -> PATH_NMI
condor_base -> PATH_CONDOR
globus -> PATH_GLOBUS

rundir -> RUN_DIR

ADMIN_EMAIL

This is the email address that is used by the NMI framework to send debug and error information to. For example, if a component of the framework crashes while executing a job, the debug information will be sent to ADMIN_EMAIL.

ADMIN_EMAIL = nmi-admin@example.com

condor_base

Defines the path to the root directory of your submit node’s Condor installation.

condor_base = /path/to/condor

NOTE: As of NMI 2.2.0, this parameter has been deprecated. Use PATH_CONDOR instead.

CONDOR_CONFIG

This paramter defines the location of the submit node’s main Condor configuration file. It is needed so that the framework can access Condor utilities for submitting, mananging, and removing builds and tests.

CONDOR_CONFIG = /path/to/condor/etc/condor_config

database

To be written

database = history

NOTE: As of NMI 2.2.0, this parameter has been deprecated. Use DB_NAME instead.

DB_HOST

This parameter is used to define the host name of the database server used by the NMI framework. It can be a fully-qualified host name or simply “localhost”. If your database server listens on a non-standard port, please use the DB_PORT parameter to define the connection port.

Prior to release 2.2.0, this parameter was named mysqlhost.

DB_HOST = database.example.com

DB_NAME

Defines the database the framework will use to read and write build and test information. Be sure that the DB_WRITER_USER and DB_READER_USER have appropriate access to this database.

Prior to release 2.2.0, this parameter was named database.

DB_NAME = nmi_history

DB_PORT

If your database server listens on a non-standard port, you must provide this port number in your NMI configuration file so that the core framework code and the default web interface can gain access. The value should be a integer without a preceding colon.

DB_PORT = 1234

DB_READER_PASS

This parameter defines the password used by DB_READER_USER to access the framework database. Please refer to the framework installation instructions on how to create users with minimal privileges.

Note that blank passwords are now allowed.

DB_READER_PASS = some_pass

DB_READER_USER

Along with DB_READER_PASS, this parameter defines the user name of a database user that has read-only access to the database information. This account is used by the NMI framework web interface and other utilities that only require read-only access to the database. Please refer to the framework installation instructions on how to create users with minimal privileges.

DB_READER_USER = some_user

DB_TYPE

This parameter is used to define what database system is used at your site. It allows the framework and web interface to use the proper database connector utility to access the build and test information.

NOTE: The only supported database this time is MySQL. Please contact the NMI developers team if you would like to deploy the framework using a different database server.

DB_TYPE = mysql

DB_WRITER_PASS

To be written

DB_WRITER_PASS = some_password

DB_WRITER_USER

Along with DB_WRITER_PASS, this parameter defines the user name of a user that has access to the NMI database. Please refer to the framework installation instructions on how to create users with increased privileges.

It is advised that you do not use the same account for DB_WRITER_USER as DB_READER_USER.

DB_WRITER_USER = some_user

DEFAULT_INPUT_HOST

Fully-qualified domain name of the central repository where the build and test artifacts and data is stored. For most installations, this value should be the same as THIS_HOST.

DEFAULT_INPUT_HOST = site.example.com

disk_thresh

This parameter defines when the disk cleaner will run if free disk space (in MB) is less than the value.

The default value is 400000 MB (390 GB)

disk_thresh = 400000

FETCH_RETRY_COUNT

This parameter became available in Metronome 2.2.4.

By default, Metronome will try to fetch an input three times before giving up. This parameter allows you to change that to another integer for the inputs in the same submit file.

The machine default may be changed in nmi.conf using the parameter FETCH_RETRY_COUNT.

globus

To be written.

NOTE: As of NMI 2.2.0, this parameter has been deprecated. Use PATH_GLOBUS instead.

main_webserver

Web server for the main entry point to the web view of the builds. This is only necessary if there are more than one submit nodes in your pool.

main_webserver = main-webserver.example.com

MAX_MATCH_WAIT

Specifies the default maximum number of seconds that a run may remain in the queue without ever being successfully matched with a resource, before it will be automatically removed. Users may redefine this value on a per-run basis via the max_match_wait attribute in their run specification file.

Defaults to six days if left undefined.

MONITOR_BACKUP_LOGS

When set to true, the monitor will make backup copies of monitor.out and monitor.err in its run directory. Without this flag, Condor will overwrite the files’ contents when the monitor is restarted. This can happen if the jobs are put on hold or Condor is restarted at the submission point. The default is set to false. There are no negative implications to not having this set to true — it is mostly used for debugging.

MONITOR_BACKUP_LOGS = 1

mysqlhost

To be written.

mysqlhost = database.example.com

NOTE: As of NMI 2.2.0, this parameter has been deprecated. Use DB_HOST instead.

mysqlport

mysqlport = 1234

NOTE: As of NMI 2.2.0, this parameter has been deprecated. Use DB_PORT instead.

nmiprefix

To be written

nmiprefix = /path/to/nmi

NOTE: As of NMI 2.2.0, this parameter has been deprecated. Use PATH_NMI instead.

password

To be written

password = some_password

NOTE: As of NMI 2.2.0, this parameter has been deprecated. Use DB_PASS instead.

path

To be written.

path = rundir

PATH_CONDOR

Defines the path to the root directory of the submit node’s Condor installation. This directory should contain the bin, sbin, and lib subdirectories for Condor. It doest not need a trailing slash.

PATH_CONDOR = /path/to/condor

PATH_GLOBUS

Path to the local Globus installation on the submit node.

PATH_GLOBUS = /path/to/globus

PATH_NMI

To be written

PATH_NMI = /path/to/nmi

PLATFORM_JOB_TIMEOUT

This configuration option became available in Metronome 2.5.1.

Sets an absolute upper limit on the length of time a platform job may run (including Metronome-internal stages). The timeout is parsed in the same way as the remote_*_timeout.

PLATFORM_TYPE

First available in Metronome 2.2.8.

PLATFORM_TYPE over-rides the hard-coded default of “nmi” for the platform type, and is in turn over-ridden by the configuration file option platform_type.

This parameter is for use in Metronome installations which use job migration. By default, Metronome uses the ‘nmi’ platform-naming scheme, which the developers found appropriate for their work. Other installations may want to name their platforms differently. Prior to Metronome 2.2.8, this would require changing the value of ‘nmi_platform’ as reported by the Hawkeye script we supply. With platform types, you can advertise more than one platform name for each machine, for instance, ‘nmi_platform’ and ‘etics_platform’. If your users prefer ETICS-style naming, you can set PLATFORM_TYPE to ‘etics’, and they won’t have to set platform_types in all of their submit files to use their preferred names.

POLLING_BACKOFF

When set to true, this parameter will cause the logfile monitor to exponentially backoff when polling a submission’s files. During a polling cycle, if there was no new information added to logfiles, the monitor will sleep twice as long as it did in the previous cycle. The max sleep time can be controlled with the POLLING_MAX variable.

The default is to disable exponential backoff.

POLLING_BACKOFF = 1

POLLING_INTERVAL

This variable allows you to control how many seconds the logfile monitor will sleep after reading the contents of a submission’s logfiles. If set to zero, the framework will continously poll the logfiles. See this page for more information on how to enable the exponential backoff feature when polling files.

The default value for this paramter is 1 second.

POLLING_INTERVAL = 1 # seconds

POLLING_MAX

Sets a limit on how much the monitor will be allowed to sleep when using the exponential backoff option. A low values will cause the logfile monitors will read the files more often to determine whether new information has been posted from the execution nodes; a high values will cause the framework to possibly update more slowly when a change occurs.

The default is 128 seconds.

POLLING_MAX = 128

protocol

This paramter defines the default protocol for fetching builds submitted by and stored on this machine.

Possible options include:

http
ftp

protocol = http

REMOTE_SITES

To be written

ROUTING_TABLE

To be written

rundir

To be written

rundir = /path/to/nmi/rundir

NOTE: As of NMI 2.2.0, this parameter has been deprecated. Use RUN_DIR instead.

RUN_DIR

To be written

RUN_DIR = /path/to/nmi/rundir

RUN_DIR_URL

To be written…

RUN_DIR_URL =

THIS_HOST

Fully-qualified domain name of the submit host.

THIS_HOST = submit-node.example.com

url_prefix

This parameter defines the relative path where the web pages are located.

For example, if the framework webpages are installed /data/www/htdocs/nmi, where /data/www/htdocs/ is the root directory of your webserver, url_prefix should be set as ‘nmi’.

url_prefix = nmi

username

To be written

username = some_user

NOTE: As of NMI 2.2.0, this parameter has been deprecated. Use DB_USER instead.

use_category_throttling

This parameter became available in Metronome 2.5.0.

This parameter may only be used with Condor 6.9.5 or later. If set to true, Metronome will use a new Condor mechanism (“category throttling”) to control the number of tasks running on the submit node at the same time for any given run. This will reduce the load on the submit node from long-running and/or io-intensive platform_pre or platform_post jobs in multiple-platform runs.

Administrators may also wish to limit the load on their submit nodes by limiting how many jobs run there at once. Condor calls these jobs “scheduler universe”, and it should possible to use load measurements in the scheduler universe’s start expression. Doing so, however, is outside the scope of this manual.

Administrators may also wish to throttle submit host load on a per-user basis (so that heavy users don’t starve others), but Condor does not yet support this.

See the Condor manual for more information.

use_condor_job_leases

Added in Metronome 2.2.8.

If this parameter is set to true, Metronome will set job leases for its platform_jobs, which makes them tolerant of service interruptions on the submit hosts. Because Metronome requires streaming output, this option can not be used prior to Condor 6.9.4.

WEBSERVER

To be written

webserver = server.example.com

Metronome Usage Accounting

Metronome Usage Report

The Metronome usage report measures the heterogeneity and frequency of builds and tests submitted by NMI Lab users, as well as summarized by project. It is generated by the nmi_usage_stats.pl script.

The report measures the number of platform_job tasks, rather than all tasks or the number of runs, because the platform_job count offers the best measure of usage. A report of runs would over-report users who submit multi-platform builds as multiple, separate runs vs. users who submit them together in a single run, even though both users performed identical operations. Likewise, a report of all tasks would over-report users who break their builds into many more granular tasks vs. users who perform identical builds monolithically in a single task.

The number of platform_job tasks, in contrast, measures the number of builds or tests a user submitted to individual platforms, regardless of whether those builds or tests were submitted as a single multi-platform run or multiple single-platform runs, and regardless of whether the builds or tests were defined as a single task, or were broken up into many smaller tasks to be individually recorded in the DB.

NOTE: this report does not measure the actual resource consumption of the user. It reports equally a five-minute or ten-hour test. For a measure of resource consumption (as opposed to heterogeneity and frequency of builds and tests), see the Condor usage report.

Metronome Web Interface

To be written…

Customize Site Branding

This page provides information on how to customize the web interface for your organization. All the changes shown below should be made in the web interface configuration file (etc/config.inc).

Site Title

To change the text used in the web browser title, as well as on the sidebar column, modify the SITE_TITLE configuration parameter.

   // -------------------------------------------------------
   // SITE TITLE
   // This is how the site will be branded
   // -------------------------------------------------------
   define('SITE_TITLE',    'New Site Title');

Sidebar Logos

By default, the web interface displays the NSF and NMI lggos in the sidebar column. This can easily be changed to either replace or add additional logos for your organization. There are three base parameters that control these logos: SITE_LOGO, SITE_LOGO_LBL, and SITE_LOGO_URL. You can remove the logos from the page by removing these parameters from etc/config.inc.

It is easy to change not only the logo image, but also its url and label on your site. For example, the logo image is defined with the SITE_LOGO parameter, and the corresponding alternative text is defined with SITE_LOGO_LBL. If you want the logo to be a link to some site, you may also define SITE_LOGO_URL as an address that the image will link to.

   define('SITE_LOGO',     'http://www.example.com/images/logo.gif');
   define('SITE_LOGO_URL', 'http://www.example.com');
   define('SITE_LOGO_LBL', 'Example Site');

To display multiple logos, add a unique numerical suffix to the end of each set of parameter starting at one. As shown in the example below, to display two logos one needs to define SITE_LOGO1 and SITE_LOGO2, along with the corresponding label and url parameters with the same numerical suffix.

   define(‘SITE_LOGO1’,     ‘http://www.example1.com/images/logo1.gif’);
   define(‘SITE_LOGO_URL1’, ‘http://www.example1.com’);
   define(‘SITE_LOGO_LBL1’, ‘Example Site #1’);

   define(‘SITE_LOGO2’,     ‘http://www.example2.com/images/logo2.gif’);
   define(‘SITE_LOGO_URL2’, ‘http://www.example2.com’);
   define(‘SITE_LOGO_LBL2’, ‘Example Site #2’);

Installing Web Interface

1) Expose the contents of this directory (‘web’) so it appears under Apache’s DocumentRoot. You can copy or move it under the DocumentRoot or just create a symlink, for example:

# ln -s /web /nmiweb

Here is the top-level of where you unpacked the NMI
framework tarball.

2) Make a copy of index.php.sample to index.php Please make sure that it has the proper permissions.

% cd /web
% cp index.php.sample index.php
% chmod 0644 index.php

You will need to edit index.php to add the path of your current directory (the one containing index.php) and the path to your NMI configuration file. Be sure to include the trailing slash in the BASE_PATH. For example,

define('BASE_PATH', '/path/to/this/file/');
define('NMI_CONF', '/path/to/nmi.conf');

would become:

define('BASE_PATH', '/home/pavlo/public/html/www/');
define('NMI_CONF', '/nmi/etc/nmi.conf');

3) Copy the sample website configuration file to make a new ‘etc/config.inc’ and tweak the settings for your site.

% cd etc/
% cp config.inc.sample config.inc
% chmod 0664 config.inc

4) Make the interface aware of where to look for .out and .err files for the builds. Let’s say your nmi.conf file has setting:

rundir = /path/to/builds

and let’s say your Apache DocumentRoot is /var/www/html. Then choose some value for the relative URL path to .out/.err files (say, “foo”). Configure it by setting

path = foo

in your nmi.conf, then create a symlink to expose the files:

# ln -s /path/to/builds /var/www/html/foo

5) Point your browser to the address of index.php Check your Apache error/access logs if the page does not load correctly You can also turn on debugging output by uncommenting the following line in index.php

//error_reporting(E_ALL);
error_reporting(E_ERROR | E_PARSE);

would become:

error_reporting(E_ALL);
//error_reporting(E_ERROR | E_PARSE);

To turn on the developers debug window at the bottom of every page, set the DEBUG attribute in index.php to true:

define('DEBUG', true);

Web Configuration Parameters

This page provides a list of the configuration parameters for the web interface. When installing the web interface, you need to copy the sample configuration file (etc/config.inc.sample) to the standard location (etc/config.inc).

DATE_FORMAT

This parameter defines the format that all dates on the side are displayed as. This is useful for international sites where the traditional date format is different than that of the United States.

See the PHP date function documentation on what options are available for this paramter.

   define('DATE_FORMAT', 'M-d-Y H:i');

To append suffixes, such as UTC, to the end of the date format, you must escape the characters you want:

   define('DATE_FORMAT', 'd.m.y H:i \U\T\C');

DEBUG_PAGE

If the web DEBUG option is set to true, this file will be displayed at the footer of every web page. It is not advised to change this parameter.

define('DEBUG_PAGE', 'msgs/debug');

ERROR_PAGE

This defines the error page that is shown to users when a critical error occurs. It is advised to not change this parameter.

define('ERROR_PAGE', 'msgs/error');

HOME_PAGE

Default homepage of the web framework. This is the page that users will be taken to when no page parameter is defined.

define('HOME_PAGE', 'results/index');

NMI_BIN

The path to the directory containing the NMI framework executables. The default is to use the bin directory in PATH_NMI.

define('NMI_BIN', PATH_NMI.'/bin/');

NMI_POOL_INFO

A temporary file with the output of condor_status -l. This is automatically created by the web framework and therefore needs to be readable/writable by the web server.

define("NMI_POOL_INFO", "/tmp/nmi-pool-info");

NMI_POOL_INFO_EXPIRE

This parameter defines how often the NMI_POOL_INFO file will be updated by the web framework. The default time is 180 seconds.

define("NMI_POOL_INFO_EXPIRE", 180);

PAGE_FOOTER

The HTML file to be included as the footer of every page. Changing this file will allow users to customize the look and feel of their site. It is not advised to change this file path.

   define('PAGE_FOOTER', INCLUDE_PATH.'pageFooter.inc');

PAGE_HEADER

The HTML file to be included as the header of every page. Changing this file will allow users to customize the look and feel of their site. It is not advised to change this file path.

   define('PAGE_HEADER', INCLUDE_PATH.'pageHeader.inc');

RESULTS_ROWS

This parameter defines the default number of rows shown in the runs and tasks listing pages. If set to zero, the results page will always default to show all records; we advise not doing this, however. The default is 20.

   define('RESULTS_ROWS', 20);

RUN_ALLOW_USER_NOTES

If this is set to true, users will be allowed to write notes and comments about runs on the details page. This feature may be useful for allowing users to keep track of why certain builds or tests have failed. Sites with older installations (Metronome release 2.2.3 and older) must make sure the DB_READER_USER account has the proper permissions. See these release notes for more information on what changes are necessary.

Due to security concerns, this feature is disabled by default.

   define('RUN_ALLOW_USER_NOTES', false);

RUN_USER_IDENTITY_COLUMN

Defines which column in the Run table to display as the “owner” of a run on the web status pages. Defaults to user (the account name of the submitting user).

If multiple people submit runs from the same user account, you can include an identity attribute in each run specification file with the owner’s “real” name, change this parameter to identity, and you will see that name displayed instead of the submit account name on the web status pages.

define('RUN_USER_IDENTITY_COLUMN', 'user');

SESSION_EXPIRE

This directive defines how long the web session cookie will be active. The expiration time is set in seconds; the default is one year.

   define('SESSION_EXPIRE', 31536000); // 1 year 60*60*24*365

SESSION_NAME

The cookie name used for the web sessions. You can change this if you would like to have multiple Metronome installations within the same domain.

   define('SESSION_NAME',   'MetronomeSessID');

SIDEBAR_WIDTH

This parameter defines the width of the sidebar column in the web interface. If no unit of measurement is provided, the default is in on-screen pixels. This may be useful to administrators that want to use an alternative PAGE_HEADER file.

   define('SIDEBAR_WIDTH', 200);

To use a relative value, such as a percentage of the total page width, you may change the parameter as shown below. Make sure to enclose the new parameter with quotation marks.

   define('SIDEBAR_WIDTH', '15%');

SITE_LOGO

This parameter defines the logo image shown in the web interface’s sidebar column. The provided image location may be either a full address (e.g., http://www.example.com/logo.gif), or a relative file path in the web interface’s public installation directory (e.g., images/logo.gif). You may also define multiple logos to be used in the sidebar by adding a numerical suffix to the end of the parameter (e.g., SITE_LOGO1, SITE_LOGO2…). The alternative text and link url for this image when it is displayed can be defined with the SITE_LOGO_LBL and SITE_LOGO_URL parameters.

   define('SITE_LOGO',     'images/nsf_logo.png');

See this page for more information about customizing your web interface installation.

SITE_LOGO_LBL

This parameter defines the alternative text label used for the SITE_LOGO image displayed on the web interface’s sidebar column. If no label is provided, the web interface will default to the SITE_TITLE parameter. The text for multiple logos may be defined by adding a numerical suffix to the end of the parameter (e.g., SITE_LOGO_LBL1, SITE_LOGO_LBL2…).

   define('SITE_LOGO_LBL', 'Example Logo Label');

See this page for more information about customizing your web interface installation.

SITE_LOGO_URL

This parameter defines the URL used for the SITE_LOGO image displayed on the web interface’s sidebar column. If no URL is provided for a logo, it will default to the homepage of the web interface. The URLs for multiple logos may be defined by adding a numerical suffix to the end of the parameter (e.g., SITE_LOGO_URL1, SITE_LOGO_URL2…).

   define('SITE_LOGO_URL', 'http://www.example.com');

See this page for more information about customizing your web interface installation.

SITE_TITLE

The branding title for the web framework installation.

define('SITE_TITLE', 'NMI Build &amp; Test System');

TABLE_FOOTER

The HTML file to be included as the footer in the closing of every table in the web interface. Changing this file will allow users to customize the look and feel of their site. It is not advised to change this file path.

   define('TABLE_FOOTER', INCLUDE_PATH.'tableFooter.inc');

TABLE_HEADER

The HTML file to be included as the header in the opening of every table in the web interface. Changing this file will allow users to customize the look and feel of their site. It is not advised to change this file path.

   define('TABLE_HEADER', INCLUDE_PATH.'tableHeader.inc');

If one would like to create a custom table header file, there are predefined parameters that can be added into the HTML that are replaced by the web framework when the page is display:

TABLE_TITLE: The title to be displayed in the table header.
TABLE_ID: A unique identifier for this table. This is used for JavaScript visibility controls.
TABLE_DISPLAY: The display style sheet attribute of the table

TABLE_VISIBILITY: The visibility style sheet attribute of the table.

URL

Defines the web accessible location of index.php. This parameter should not need to be changed as it is pre-configured to set the URL from the web server’s environment information and the WEBSERVER parameter.

   // -------------------------------------------------------
   // URL
   // This URL should point to the location of the index.php file.
   // -------------------------------------------------------
   define('URL', 'http://'.WEBSERVER.$_SERVER["PHP_SELF"]);

USE_PLATFORM_LIST_CACHE

On large installations, there is an issue with retrieving the list of unique platforms for runs. By enabling this parameter the system will always read from a static file instead of querying the database.

NOTE: This should always be set to false for all sites except the University of Wisconsin-Madison. Future versions will allow users to use this option.

define('USE_PLATFORM_LIST_CACHE', false);

VERSION_FOOTER

When this value is set to true, the web framework will display the framework version and page render time at the bottom of the all web pages. Default is true.

define('VERSION_FOOTER', true);

WARNING_MESSAGE

This parameter defines the warning header shown to users when a non-critical error occurs in the web interface. It is advised to not change this parameter.

Note: This paramter has been deprecated. See WARNING_PAGE

   define('WARNING_MESSAGE', INCLUDE_PATH.'warning.inc');

WARNING_PAGE

This is the file that is loaded to display warning messages in the web interface. It is not advised to change this parameter.

   define('WARNING_PAGE',  'msgs/warning');

Notes

Installation tasks:

Install ports used for easier administration

cd /usr/ports/sysutils/portupgrade
make install
portinstall sysutils/cfengine2
portinstall net/cvsup-without-gui
portinstall net/openldap-client
portinstall net/pam_ldap
portinstall net/nss_ldap

Install Java

Note that this must be done by hand because you must download the files directly from Sun. See http://www.freshports.org/java/jdk14/ for more information
portinstall -m "-DMINIMAL" java/jdk14

x86_macos_10.4

Notes

As of 2006-07, this platform should be configured to use an old version of Condor that does not have a bug in procapi, e.g., Condor 6.7.8.

x86_rh9

Required

User Setup Instructions

Using the framework

Once you have installed and configured the NMI software, create a job
submission file, called, say (see below for references
on the syntax and examples) and submit a build as follows:

export NMI_BIN=/bin NMI_LIB=/lib
source config.sh
nmi_submit

Miscellaneous notes

IMPORTANT: You should source config.sh (resp config.csh, depending on
your shell) to have your PATH and PERL5LIB environment variables set up
correctly, before you start using the NMI software.

The main command for submitting build or test jobs is located at
<prefix>/bin/nmi_submit.

See http://www.cs.wisc.edu/condor/nmi/builds/usersguide for examples
of NMI command files and glue.

If you have multiple users submitting builds, you should give the
rundir directory from nmi.conf (say, named /nmi/run) permissions
to enable builds to be owned by their respective users:

chmod 777 /nmi/run
chmod o+t /nmi/run

Using SoftEnv to Manage Prerequisite Software

Metronome has built-in capabilities to utilize SoftEnv managed prereq software on computing resources, as an alternative to Metronome’s native prereq mechanism. Both the native NMI prereq mechanism and softenv-managed prereqs can coexist in the same pool, and even be used together for different prereqs in the same build or test run — there need be no visible difference to Metronome users, only admins.

When a build or test is submitted to an execution machine, Metronome will first look in the SOFTENV_ROOT environment variable for the base location of SoftEnv, or if it is undefined, will attempt to pull the location from the PATH environment variable. Users do not need to add any additional parameters to their submission files.

Configuring Hawkeye

Follow the installation instructions on how to install and configure the Hawkeye modules. The only change that needs to be made is that the NMIPOOL_SOFTENV_ROOT parameter needs to point to the base installation directory of the SoftEnv software package:

NMIPOOL_SOFTENV_ROOT = /prereq/softenv-1.4.2

Command-Line Tools

disk_cleaner.pl

Since build and test results are archived on the submit node after each run completes, it is necessary to periodically purge them to prevent the disk from filling up. The NMI software now comes with a disk cleaner that removes build results that are older than a given number of days. It is installed along with the rest of the NMI executables in $PREFIX/bin. The executable’s name is disk_cleaner.pl

Usage

disk_cleaner.pl [options]

Options

--days=
Required.

--dry-run
Generate the list of directories that will be cleaned but do not actually
delete them. No email is sent in this case.

--find-trash
Find tarballs sitting around in builds that have been marked as clean.

--force
Forces the cleaning part to execute even if there is enough disk space.

--help
Usage information.

--notify=
Optional. Defaults to uw-nmi-dev.cs.wisc.edu.

--project=
Optional. Clean builds from this project only.

--quiet
Suppresses output unless an error occurs.

--user=
Optional. User whose runs need to be deleted.

--skip-runids=

Example

At UW-Madison, we use roughly the following line in root’s crontab on our submit nodes to run the cleaner daily at 1am and remove any builds older than 3 days:

0 01 * * * /full/path/to/disk_cleaner.pl --days=3 --notify=nmi-systemcs.wisc.edu@

nmi_migrate_run

This utility allows an administrator to transfer a run’s archived output (aka the “rundir“) from the localhost to a different remote-host. The web status pages and other Metronome utilities will discover this new location without any additional configuration. Note that when a run’s archive is transfered to a different host, its GID does not change (it continues to include the hostname of the original submission host in its text string).

If this tool is executed as root, it is able to transfer any run archives; otherwise, it will only allow a user to transfer their own archives. It also assumes that the user account on remote-host exists and matches that on the localhost.

There are six main steps when migrating an archive to a new remote-host. If any step fails along the way (except for removing the original archive), the process will be aborted and all changes are reversed.

Create a tarball of the original run directory on the localhost
Create a new run directory on the remote-host
Transfer the tarball to the remote-host
Unpack the tarball into the new run directory on the remote-host
Update the database with the new submit host and filepath information

If the “—remove-local” is provided, delete the original run directory on localhost

By default, the tarballs are stored in /tmp/ on both the localhost and the remote-host; this can be overridden using the “—remote-temp” and “—local-temp” commandline options. Likewise, the new run directory on the remote-host will be installed in the base directory provided in the RUN_DIR configuration parameter; this can also be overridden using the “—remote-dir” option.

In this example, nmi_migrate_run will transfer a run to a different remote host (assuming SSH keys have already been setup). If SSH keys have not be configured properly, the utility will prompt the user multiple times for their remote login password on the remote-host. The “—remove-local” will remove the directory only after the run directory has been successfully transfered and verified, and if the database has been properly updated.

$ nmi_migrate_run --remove-local
Successfully migrated XXXX_hostA.cs.wisc.edu_1149795301_9046 to hostB.cs.wisc.edu

The nmi_migrate_rm utility can also be used to move old run directories (that is, jobs submitted before Metronome 2.2.4) to the directory hierarchy format. This is essentially the same process as in the previous example, except that remote-host is the same as the THIS_HOST parameter in your NMI configuration file.

Usage: nmi_migrate_run [options] remote-host [runid|gid]...
        --nmiconf=    Select which NMI configuration file to use
        --local-temp   Local temp directory to store files
        --remote-temp  Temp directory on remote-host to store files
        --remote-dir   Directory on remote-host to store runs. Default is RUN_DIR
        --remove-local      Remove local run directory after migrating
        --db-only           Only update the database
        --help              Print this message

Setting Up SSH Keys

In order to allow the nmi_migrate_run tool to run automatically and batch multiple runs, it is advised to setup SSH keys for password-less logins between the hosts you are transferring jobs to. This is a brief step-by-step guide on how to set this up at your site. In the example below, the SSH keys are setup for one particular user. If you need to migrate multiple users jobs, you may want to setup keys for the root account and transfer all the jobs using this privileged account.

This process may not work properly if your hosts use Kerberos or AFS. Please contact your site administrator if you are having problems.

First, on the localhost where the run directories are originally stored, create SSH keys with the ssh-keygen command. When prompted for a passphrase, just hit enter for no phrase:

[hostA]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/fakeuser/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/fakeuser/.ssh/id_dsa.
Your public key has been saved in /home/fakeuser/.ssh/id_dsa.pub.
The key fingerprint is:
 fakeuser@hostA

Next, transfer the id_dsa.pub file to the remote-host you want to transfer jobs to:

[hostA]$ scp /home/fakeuser/.ssh/id_dsa.pub fakeuser@hostB.cs.wisc.edu:~/hostA.key

Now add the contents of this file to the authorized_keys file for the same user on the remote-host: Make sure that this file has limited reading permissions:

[hostB]$ cat ~/hostA.key &gt;&gt; ~/.ssh/authorized_keys
[hostB]$ chmod 0600 ~/.ssh/authorized_keys

Lastly, test whether the keys work and you can login automatically:

[hostA]$ ssh nmi-s003
Last login: Thu Jun  7 15:21:37 2007 from hostB.cs.wisc.edu
[hostA]$

nmi_usage_stats.pl

This script is located in the bin dir of Metronome install and requires the Mail::Mailer Perl module (perl -MCPAN -e 'install Mail::Mailer'). Use it as:

export PERL5LIB=$NMI_BIN;$NMI_BIN/nmi_usage_stats.pl --days=1 --mailto=your-email@your-domain

Adjust the number of days as needed. Add it to cron to run on a daily and/or weekly basis if preferred.

Use a submit node or an NMI machine with an nmi.conf file set up.
Install

Reference Manual

Introduction

Metronome Installation

Preparing System Environment

Preparing the Database

Installing NMI Framework

Framework Configuration

Hawkeye modules setup

Advanced Hawkeye Usage

Running publish_dir.pl

Configuring a Metronome Pool for Parallel Testing

Overview

Version Requirements

Pool Setup

Configuring jobs to require certain hosts, or visa-versa

Additional Notes

What is =?= and How Is It Different From ==?

Debugging

Cross-Site Job Migration

Overview

Limitations

NMI Configuration

NMI Configuration File

Remote Sites List

Submit Node Configuration

Cron Job

Firewall Configuration

Setting Condor’s Listening Port Range

Local Inclusion of Remote Resources

Remote Worker Nodes Connecting to Local Collector

Remote Submit Nodes Connecting to Local Resources

Local Central Manager Connecting to Remote Resources

Job Migration

Metronome Configuration Parameters

ADMIN_EMAIL

condor_base

CONDOR_CONFIG

database

DB_HOST

DB_NAME

DB_PORT

DB_READER_PASS

DB_READER_USER

DB_TYPE

DB_WRITER_PASS

DB_WRITER_USER

DEFAULT_INPUT_HOST

disk_thresh

FETCH_RETRY_COUNT

globus

main_webserver

MAX_MATCH_WAIT

MONITOR_BACKUP_LOGS

mysqlhost

mysqlport

nmiprefix

password

path

PATH_CONDOR

PATH_GLOBUS

PATH_NMI

PLATFORM_JOB_TIMEOUT

PLATFORM_TYPE

POLLING_BACKOFF

POLLING_INTERVAL

POLLING_MAX

protocol

REMOTE_SITES

ROUTING_TABLE

rundir

RUN_DIR

RUN_DIR_URL

THIS_HOST

url_prefix

username

use_category_throttling

use_condor_job_leases

WEBSERVER

Metronome Usage Accounting

Metronome Usage Report

Running `publish_dir.pl`

What is `=?=` and How Is It Different From `==`?