Overview

This feature of Metronome builds on the Condor Parallel Universe and provides for running jobs on multiple machines simultaneously. Condor’s Chirp mechanism makes communication between the machines possible.

Submitting parallel jobs

This node describes how to set up your pool to run parallel jobs. The UW NMI pool already has the DedicatedScheduler set up. The submitter is nmi-s005.cs.wisc.edu.

A Metronome submit file for a parallel job looks similar to a normal submit file, with the following exceptions:

• Add parenthesis around your platform list. Note that at the present time you are

only allowed one set of parens per cmdfile. The parallel nodes are assigned based on the location of the platform name in this list. In the below example, x86_fc_2 is node 0.

platforms = (x86_fc_2, x86_rhas_3)

• For any of the following task hooks that run on the remote side, enter a glue script for each node as detailed below. The platform_pre task must be defined, even if it uses a noop script. This is a limitation of the Metronome framework and will be fixed in a future release. The example assumes the glue scripts are stored in directories named glue, client and server.

pre_all = glue/pre_all


	
platform_pre_0 = client/platform_pre

platform_pre_1 = server/platform_pre

	
remote_declare_0 = client/remote_declare

remote_declare_1 =  server/remote_declare

	
remote_pre_0 = client/remote_pre

remote_pre_1 = server/remote_pre

	
remote_task_0 = client/remote_task

remote_task_1 = server/remote_task

remote_task_args_0 = 7000

remote_task_args_1 = 7001

remote_task_timeout_0 = 20

remote_task_timeout_1 = 20

remote_post_0 = client/remote_post

remote_post_1 = server/remote_post

	
platform_post_0 = client/platform_post

platform_post_1 = server/platform_post

	
post_all = glue/post_all

Communication between parallel job nodes

Chirp facilitates the underlying communication between nodes of a parallel job. Use the nmi_putattr and nmi_getattr scripts as described here to inject params directly into the job ad from one node and retrieve them from another. The scripts are sent to remote machines and are located in the NMI_BIN directory on each remote job node.

The older method of Chirp communication may be used as well. You may send a file to the head node using Chirp and then retrieve it on other nodes. Use the nmi_putfile and nmi_getfile scripts as described here for this purpose.

Metronome-provided environment variables

Certain attributes are published to the job ad and the remote environment. These attributes may be helpful in job synchronization and inter-job dependencies.

• Hostname

NMI_NODE_0_HOSTNAME=nmi-build26.cs.wisc.edu
NMI_NODE_1_HOSTNAME=nmi-build21.cs.wisc.edu

• Start time, return code and end time

NMI_NODE_0_START_remote_task=1177970566


	
NMI_NODE_0_START_remote_pre=1177970565

NMI_NODE_0_RVAL_remote_pre=0

NMI_NODE_0_END_remote_pre=1177970566

	
NMI_NODE_0_START_remote_declare=1177970563

NMI_NODE_0_RVAL_remote_declare=0

NMI_NODE_0_END_remote_declare=1177970564

• Other useful variables

NMI_BIN=/home/condor/execute/dir_13377/bin
_CONDOR_SCRATCH_DIR=/home/condor/execute/dir_13377
_CONDOR_PROCNO=0

Finding Condor CPU slots

% condor_status -const 'DedicatedScheduler=!=Undefined'

...and it will list all the “P” slots. To confirm which schedd queue each slot is bound to, run:

% condor_status -const 'DedicatedScheduler=!=Undefined' -format '%s\t' Machine -format '%s\n' DedicatedScheduler