What is Metronome?

Metronome (formerly The NMI Build & Test System) is a distributed, multi-platform framework designed to provide automated software building and testing capabilities to a variety of grid computing projects.

We believe that software isn’t reliable unless it’s regularly built and tested. Doing so requires not only a significant number of CPU cycles, but often a variety of unusual and difficult-to-maintain platforms, and a framework for automating, tracking, and monitoring the entire process.

Our goal is to provide an implementation of this framework utilizing proven grid computing tools as a foundation, as well as to support the growing number of Metronome Facilities internationally, including our own NMI Lab at the University of Wisconsin-Madison.

Our LISA 2006 paper below provides more details on the framework and its implementation, including how it differs from some other common build or test frameworks.

Terminology

Unfortunately, we (the Metronome developers) have been inconsistent over time in our own terminology for some aspects of the Metronome framework and software. This may be reflected even on this site, although we’re working to make our documentation and reference materials more consistent.

For the time being, the following is the most common vocabulary used for the Metronome workflow:

A Metronome “Run”

A user submits a Metronome run to execute a build or test workflow on one or more platforms.

A run is described in a run specification file which is passed to nmi_submit.

A Metronome “Task”

Each run consists of a number of Metronome tasks whose individual success/failure/duration/exe-host/output are tracked and recorded in the DB.

Although the actual operations performed by each task are based on user-provided scripts or executables, most tasks have a predefined name and location in their run’s workflow. Users may, however, optionally declare an arbitrary number of custom-named tasks to run on each specified platform.

In addition to recording the results of individual tasks in a run, Metronome also records the success/failure/duration of certain meta-tasks. Each meta-task represents the collective result of a predefined set of related tasks in the run. A platform_job meta-task for each specified platform in the run represents the collective success or failure of all the tasks which executed on that platform. Additionally, if users choose to declare their own custom-named tasks on a platform, the collective success or failure of just those user-defined tasks is recorded in a remote_task meta-task.

A Condor Job

All platform-independent tasks in a run are executed on the submit host by Metronome inside their own individual Condor jobs; all tasks specific to a given platform are executed on a remote machine inside a single Condor job. All the jobs used to execute a run are represented in a single Condor DAG.

Overview

A build & test run generates a number of tasks to be executed on the submission machine and one or more user-defined platforms. The diagram below shows the overall workflow.

Fetch Tasks – Retrieves needed software inputs from one or more sources to the submission machine.
Pre Run Tasks – Lightweight tasks to be performed on the submission machine in order to prepare the software for staging to the remote platforms. These tasks can be global to all platforms (and thus executed only once, on the common input data) or specific to each platform (and thus executed once per platform, on its specific copy of the input data).
Remote Tasks – Tasks to be performed on each remote platform.
Post Run Tasks – Lightweight tasks to be performed on the submission machine to manipulate the results of the remote tasks. These tasks can be specific to each platform (and thus executed once per platform, on its specific results), or global to all platforms (and thus executed only once, on the combined output).

Task Hooks

The Build & Test System organizes a build or test workflow into predefined stages, or tasks. Each one provides a “hook” where you can optionally define a custom script or program to execute that task in the execution process. Only the remote_task task is required. The diagram below shows all of the available task hooks:

User-Defined Sub-Tasks

The Build & Test System gives you the option to divide the platform-specific remote_task into any number of user-defined sub-tasks. The diagram below shows where these user-defined tasks appear in the execution process:

Task Failure Handling

The build and test system handles failures in a way that allows the user to detect what went wrong. The handling depends on the task. The various ways that failures are handled are as follows:

• abort run – Fatal failure of the entire run. No subsequent tasks are executed after the failure.
• abort platform – Fatal failure of one platform. No subsequent tasks specific to the given platform are executed after the failure. Once the platform-specific tasks for all other platforms complete, the run is aborted.
• continue remote / abort platform – The failure is recorded, but subsequent remote tasks continue. When all the remote tasks for the given platform complete, the platform is aborted.

• remote_post / abort platform – Run skips remaining remote tasks and “jumps” to remote_post. This allows the creation of a results.tar.gz file for analysis. After remote_platform returns, the platform is aborted.

Timeouts

Timeouts may be implemented as run or task specific.

Run specific timeouts refer to the amount of time a Condor job is alive in the queue before being removed. The automated removal assists with cleaning up jobs with mismatching requirements, or jobs which will never run due to various system problems. The default is 6 days, which may be overridden. The order of execution is the run spec file (max_match_wait), then the NMI config file (MAX_MATCH_WAIT), then the default value. Timeout values are specified in seconds.

Task specific timeouts are set in the NMI submit file. These timeouts are typically set by users at task boundaries to assist with shutting down services at the correct time or to prevent services from hanging indefinitely if something should go wrong upon shutdown. Please see the appropriate section of the manual for details.

Build/Test Specification File Commands

The following commands are available in the NMI submit file (aka the build or test specification file). The syntax for each command is:

Any whitespace before or after the command or value is ignored (however whitespace within a value is retained).

Certain special variables in the file will be expanded before processing.

Input Specification File Commands

Each build or test specification file must reference one or more input specification files (via the inputs keyword) which define the inputs to the build or test run.

Input specification files follow the same basic format as the build or test specification file itself. The syntax for each line in the file is:

Any whitespace before or after the command or value is ignored (however whitespace within a value is retained).

Certain special variables in the file will be expanded before processing.

Output Specification File Commands

In the nmi_submit submit file, one may define “outputs” to instruct the system to transfer some or all results to an external repository after the run is complete.

The syntax is as follows:

outputs = output_file_1, output_file_2, ...

Currently, the following output methods are supported:

• scp

• gridftp

Some example output files:

method = scp
platform = x86_slc_3
source = results.tar.gz
dest = /tmp/tolya

or

method = scp
platform = common
source = results.tar.gz
dest = /tmp/tolya-common

(in the last case, it is user’s resposibility to create the results file
under common/)

Here’s an example of using gridftp method:

method = gridftp
platform = x86_rh_9
source = results.tar.gz
dest = my.gridftp.host/data/rh9/results.tar.gz

Variable Substitution Inside NMI Specification Files

It is possible to perform macro substitution within the NMI run specification file and input specification files. There are two substitutions done at submit time:

1. If you specify $(USER), it will be replaced by your login name.

1. If you specify any other macro $(FOO) in your NMI run specification file, it will be immediately replaced by the value of the environment variable $_NMI_FOO, if defined in the submitter’s environment. (Otherwise it is replaced by the null string.)

nmi_condor_status

nmi_condor_status prints Metronome-specific information about your Condor pool. It accepts three options, in addition to the usual --nmi-conf. The -w, -ww, and -a options control the width of the printed information. Normally, nmi_condor_status prints output in 80 columns, truncating fields to fit. The -w and -ww print more columns; the -a option prints unlimited-width and unjustified output more-suitable for use by scripts.

For advanced users, starting with Metronome version 2.2.3, you may configure your pool so that nmi_condor_status ignores certain startds. This is useful if your pool includes non-Metronome hosts, or hosts which will never run jobs for other reasons. (For instance, if you’re running Hawkeye to monitor a submit host.) Setting the attribute NMI_isExecHost to FALSE in your Condor configuration file and adding NMI_isExecHost to STARTD_EXPRS will cause nmi_condor_submit to ignore that startd.

nmi_gid2runid

This command has been available since before Metronome 2.2.2.

This command converts Metronome GIDs (strings) to Metronome runids (integers). It takes one argument, the GID, and one option, the ubiquitous —nmi-conf, which allows you to set the location of the Metronome configuration file:

If the database is down when you submit a job, this command can be used at a later date to determine the runid, as a command-line alternative to searching via the web interface.

nmi_pin

Command used to tell the build and test system database to save the results of a run beyond the usual timeframe. Here are some examples:

nmi_pin -list

lists all of the runs that are currently pinned

nmi_pin --runid=24630

Stores the record of run 24630 for the default 60 days.

nmi_pin --unpin --runid=24630

Undos the pin.

nmi_pin --runid=24630 --days=100

Stores the record of run 24630 for the default 100 days.

nmi_resource_advertiser

Usage: nmi_resource_advertiser
        --nmiconf= Select which NMI configuration file to use
        --routing-table  Prints out the routing table
        --broadcast      Broadcast resource information to all hosts
        --debug          Enable debug output

Note: when the routing table gets updated by nmi_resource_advertiser, the job router must be reconfigured in order for the new information to get loaded. In some situations, it may reconfigure all of its host’s local Condor daemons. (It can’t use the target parameter for condor_reconfig because that tool doesn’t have logic to send a reconfig command to an arbitrary daemon. The resource advertiser actually tries to be smart about updating the job router: first it tries send a SIGHUP with a killall condor_schedd.v7, then it tries a kill . If both of these measures fail for whatever reason, it then calls the condor_reconfig.)

nmi_resubmit_run

This command became available in Metronome 2.5.0.

nmi_resubmit_run use the Metronome database to recreate and then submit a new copy of a previously-submitted run. It can not recreate runs submitted by versions prior to Metronome 2.5.0. (However, at the time of this writing, we are working on a tool which will remove this restriction (on a run-by-run basis) for runs whose run directories still exist.) For example:

nmi_resubmit_run 75549

will submit a copy of runid 75549, reporting as nmi_submit normally does. By default, nmi_resubmit_run does not keep the submission directory around after nmi_submit succeeds. You may override this with the --submit-dir option, passed as a flag (the directory name defaults to the id of the run being recreated) or with an argument (specifying the submit directory’s name).

We intend this command to simplify disk-space managemenmt, as reproducing a user’s run no longer requires a run’s run directory to be preserved. We do not, however, recommend that you begin removing run directories immediately after upgrading, as this is a new feature, and may yet have bugs.

nmi_rm

The nmi_rm command can be used to remove a run from the Metronome queue. It will not remove any information from the Metronome database. Unless the invoking user is root, it will only remove jobs that are owned by you.

Here are some examples:

Usage: nmi_rm [options] [runid|gid] [runid|gid]...
        --user        Which user to remove runs for (defaults to current)
        --all               Remove all runs for the current user (or all runs if root)
        --force             Always try to remove a run and update the database
        --db-only           Only update the database
        --remove-consumers  Remove any runs that depend on a given run
        --help              Print this message

nmi_rm 24630

Kills run with a runid of 24630.

nmi_rm mbletzin_grandcentral.cs.wisc.edu_1155560480_32710

Kills a run using the GID .

nmi_rm 24630 mbletzin_grandcentral.cs.wisc.edu_1155560480_32710 24631

The gids and runids can be mixed together and given in succession on the commandline

nmi_rm --user=jsmith --all

Kills all of the runs for user jsmith

nmi_rundir

This command first became available in Metronome 2.5.1

Converts runids or GIDs given on the command-line into hostname:full-path pairs.

nmi_runid2condor

This command became available in Metronome 2.2.4.

This command searches the Condor job queue for Metronome runs with the given runid or GID. It accepts one argument, the runid or GID, and a number of options.

The --history option will check Condor’s history files for the given runid or GID.

The --global option will check all job queues known to the local machine’s Condor collector, rather than just the local machine. For example, it will check the queues of nmi-s001.cs.wisc.edu and nmi-s005.cs.wisc.edu if invoked on nmi-s003.cs.wisc.edu.

nmi_runid2condor also accepts the usual --help and --nmiconf options.

This command is generally most useful in conjunction with condor_q’s analysis flags:

In this case, run 72313 can only run on one three “machines” in the pool, two of which reject it, the other of which is busy. You can look at the web interface’s pool status pages to discover that this means a single physical hppa_hpux_11 machine exists, but this BUILD won’t be run by a TEST or PARALLEL slot, and that the BUILD slot is busy. In other cases, you might discover a typo in your prereqs (which would look something like 2 ( target.has_java_1_5_0_02 isnt undefined )0)), or that you’ve requested a combination of existing prereqs not matched by any single machine. (In which case you should probably contact your Metronome installation’s administrator!)

The Condor ID passed to condor_q was the last one listed by nmi_runid2condor. This will generally but not always be the right ID to pass — Metronome runs each map to more than one Condor job, and those jobs are not the same throughout the run’s lifetime. As in the example, however, there should be four Condor jobs for the bulk of the run. Passing the wrong Condor ID to condor_q is harmless, however, and will generally result in it saying the the job is already being serviced.

nmi_runid2gid

This command has been available since before Metronome 2.2.2.

This command converts Metronome runids (positive integers) into Metronome GIDs (strings). Metronome uses GIDs internally, so that it can run jobs during database failures. (The information is recorded on disk and entered into the database when it comes back up.) It takes one argument, the runid, and one option, the ubiquitous --nmi-conf, which allows you to set the location of the Metronome configuration file:

This command is useful when interacting directly with Condor, because the GID of a Metronome run is entered in the ad for all of its tasks. However, nmi_runid2condor converts directly from runids to Condor IDs if a job is in queue, which can be more convenient.

nmi_submit

Usage: nmi_submit
        --nmiconf=   Select which NMI configuration file to use
        --must-match       Job must match with resources before submitted
        --notify-fail-only Only send notification if job fails
        --verbose          Enable verbose output
        --quiet            Do not print job submission information
        --timeout=    Number of seconds to wait for runid. Default is 180
        --no-wait          Do not wait for runid. Program returns immediately
        --debug            Enable debug output
        --help             Show this information

This command is used to start B&T runs. The command expects a submit file as input. For example the following executes the submit file perlHelloWorld.submit located in the current working directory:

nmi_submit perlHelloWorld.submit

Although the submit file, and any input specification files it references, must exist when nmi_submit is invoked, they are not read again, and may be removed afterwards. A copy of the input files, along with all of the submission’s runtime data and output files, is archived in the “run directory”. The current working directory of the original submission has no significance.

nmi_testsforrun

This commands returns all the NMI test runids that use a given build submission as input. Given either a gid or a runid, the command prints out all the test runids one-by-one on separate lines.

Command usage:

Using a runid:

Using a gid:

NMI-Provided Environment Variables

nmi_getattr, nmi_putattr

Primarily intended to enable simple communication between parallel tasks on different hosts, the nmi_getattr and nmi_putattr tools are available on the remote execution platform and allow a task to read or write individual attributes from a common classad-like hash table.

Usage:

nmi_putattr attr value

_nmi_putattr_ will set the value of the given attr (overwriting any preexisting value)

returns 0 upon success
returns non-zero upon failure to set the attr for any reason

nmi_getattr attr

_nmi_getattr_ will print to stdout the value of the given attr.

returns 0 if the attr is defined
returns 1 if the attr is undefined
returns >1 upon failure

Note: These tools are non-blocking. nmi_getattr will return 1 if you try to get a value that does not exist. The user must poll for values and determine how long to wait for a value to exist before giving up and declaring failure. Future plans for a Metronome-provided polling mechanism are in the works.

nmi_getfile, nmi_putfile

The nmi_getfile and nmi_putfile tools are used to send files between nodes of a parallel job. These scripts are available on the remote execution platform and allow a node to send or retrieve files to each other via the submit host.

Usage:

nmi_putfile local_file remote_file

_nmi_putfile_ will send a file to the submit host

returns 0 upon success
returns 1 if the chirp invocation fails
returns 2 if the local or remote file are not defined, or if the local file does not exist
returns actual error code >1 upon failure

nmi_getfile remote_file local_file

_nmi_getfile_ will fetch a file from the submit host

returns 0 if the fetch worked successfully
returns 1 if the chirp invocation fails
returns 2 if the local or remote file are not defined
returns actual error code >1 upon failure

GIDs and Run IDs

The GID, or Globally Unique ID, is an alphanumeric identifier used by the build and test system to refer to a specific build or test submission. It is generated at submission time, and is used internally as a component of the run’s archive directory (or run directory) path.

The Run ID, or runid, is a short numeric identifier also used by the build and test system to refer to a specific build or test submission. It is only generated once the submission is successfully registered into the database (it is the primary key of the record), and as a result it may not yet be known at submission time if the database is unavailable or unresponsive.

There is a one-to-one correspondence between Run IDs and GIDs; the only practical differences are that GIDs are long and unwieldy, but are guaranteed to be known at submission time, whereas Run IDs are short and convenient, but cannot be presumed to exist until the database is successfully initialized for each run. GIDs are also unique across NMI pools, whereas Run IDs are only unique within a given NMI pool. Although having two similar identifiers can be confusing, the distinction is important and exists in order to ensure that build & test jobs can be submitted even in the face of database performance or availability problems.

GIDs may be used to look up Run IDs (if known), and visa-versa, via the nmi_gid2runid and nmi_runid2gid tools.

Both the Run ID and GID are used (often interchangeably) as arguments for many NMI command-line tools (e.g., nmi_pin). The Run ID is also used as the identifier for the nmi input method.

input specification file

• the file containing instructions for fetching a single input to a run

meta-task

In addition to the status and outcome of each individual user-defined task in a run, the Metronome DB also contains the outcome of a handful of special meta-tasks for each run. These meta-tasks represent the collective result of a set of related tasks, and are useful for reporting their outcome as a whole.

For example, the platform_job meta-task represents the collective outcome of all the tasks which executed on a given platform in a run; likewise, the remote_task meta-task represents the collective outcome of all the user-defined tasks that were declared on a given platform (however, if no user-defined tasks were declared at all, remote_task is a regular task and not a meta-task).

platform_job

For each specified platform in a Metronome build or test run, the DB records the outcome of a special platform_job meta-task representing the collective success or failure of all the tasks which executed on that platform.

run

aka build run
aka test run
aka build/test run
aka NMI run

A distinct build or test submission to Metronome (via nmi_submit). For more information, see Terminology.

run specification file

aka build specification file
aka test specification file
aka build/test specification file
aka NMI specification file
aka NMI submit file

• the file provided to nmi_submit containing instructions for running a given build or test run.

Archived Run Results

The metadata of each build/test run, and of all its tasks, are stored indefinitely in the NMI Build/Test database, so you should always be able see (or query) the outcome of past builds.

However, the user-level output data (results.tar.gz files) of build/test runs are only archived on the submission host for a limited period of time after the run has completed, as defined by each individual NMI Build & Test Facility’s policy. After that time, the data may be deleted, unless they are explicitly “pinned” by the owner using the nmi_pin command. Pinned runs are never deleted.

Unlike the database, the output data archive is typically not backed up, and is not guaranteed to exist for any length of time — so if your build or test output data is difficult to reproduce, or you need access to it indefinitely, you should copy it to your own reliable storage. To make such copying easier, we plan to add optional “put” steps that can be defined at the end of a run (analogous to the “fetch” steps), which will allow you to specify at submit-time where & how you’d like your results transferred off-site. For more information, see http://grandcentral.cs.wisc.edu/nmi_drupal/?q=node/192.

Environment Variables

A variable in the Build and Test System is any value that is set in the execution environment of a task by the system. For example, the variable NMI_PLATFORM can be accessed by remote programs in the following ways:

Meaningful Filenames

The build and test system has several well-known filenames that it looks for. These files can be used to change the behaviour of the run.

Run Directory

Run Directory

This section discusses the contents of the run directory. The path name of this directory contains the GID of the run.