Navigating the Build and Test Overview Page

Overview Home.

As shown in the figure below, when the Build & Test Overview page is first displayed, the page list all of the submissions that have been run on the submit machines. The first thing that needs to be done is to filter the submissions down to a managable number using the search box. In this case the page displays 1109 submissions.

Sort Buttons.

The submission results can be sorted by each column listed in the page that contains sort buttons. As shown in the previous picture, each column which can be sorted has two buttons which will sort the results in either descending or ascending order.

Search Box.

The search box is used to filter out unwanted submissions. The figure below shows the fields that can be used to filter the results:

1. Search all fields in the results database for the keyword.
2. Show only submissions by selected user.
3. Show only submissions of a selected run type; usually either build or test.
4. Show only results from the selected platform.
5. Advanced Search is not implemented at this time.
6. Show only the submission associated with the selected run id.
7. Show only results from the selected submit machine.
8. Show only results from the selected project. This the value associated with the project submit file command.
9. Show only results from the selected component. This the value associated with the component submit file command.
10. Show results which are Running, Completed, Failed, or Removed.
11. Show results from submissions run on the selected date.

1. Show only pinned results when selected.

Details Page

Details Page.

The following picture shows an example details page for the build and test run:

1. Clicking on this link will drop you into the run directory.
2. These two buttons point to the output and standard error of the CVS command used to stage the component.
3. This column shows the platforms the submission was run on in addition to the submit machine.

1. This column contains links for each machine used in the run. These links display a host page which lists among other things the prerequisites that are installed.

Hello World Using CVS Input

Hello World Using CVS Input

This tutorial demonstrates a simple build and test run using the CVS input method. Download the attached files and move them to the working directory of your submit machine and then follow the sections below. The first two pages discuss the contents of the files you have downloaded. This is followed by the procedure to execute the run and examine the results.

Hello World Using SCP Input

Hello World with SCP Input.

This tutorial demonstrates a simple build and test run using the SCP input method. Download the attached files and move them to the working directory of your submit machine and then follow the sections below. The first two pages discuss the contents of the files you have downloaded. This is followed by the procedure to execute the run and examine the results.

Examining Build & Test Run Results

Examining Build & Test Run Results.

Here are two ways to look at the results of a build and test run. The first takes a look at the run results directory on the submit machine. The second way uses the build and test overview web page.

CVS Tag Procedure

1. Re-use the input specification and build and test specification files from the Hello World Tutorial.

1. For this exercise we will add a cvs tag that corresponds to a hello world script with a slightly different output. Open the file perlHelloWorld.cvs in your favorite editor and add the following line:
cvs_tag = helloBranch
2. Run nmi_submit using these files.
bash$ nmi_submit perlHelloWorld.submit
3. Locate your run’s details page using the overview page with the skills you learned here.

1. Click on the output of fetch.perlHelloWorld.cvs. You should see something like this:

1. Notice in the output of the command above where the tag name is used.

1. Click on one of the remote_task outputs. The output should contain the phrase (Tagged Version) which shows that the task has run the tagged version of the script.

FTP Procedure

1. Re-use the build and test specification file from the Hello World Tutorial.

1. Download the attachment below. This is the input specification file that defines the ftp fetch. It contains the following:

1. Edit the file perlHelloWorld.submit with your favorite editor. The input needs to be changed to specify the new ftp input specification file. Change the line inputs = perlHelloWorld.cvs to inputs = perlHelloWorld.ftp.

1. Run nmi_submit using these files.
bash$ nmi_submit perlHelloWorld.submit
2. Locate your run’s details page using the overview page with the skills you learned here.

1. Click on the output of fetch.perlHelloWorld.ftp. You should see something like this:

1. Click on the outputs of the remote_task tasks. You should see the hello world from the script included in the tar archive.

Build & Test Submit File Dissection

Tasks Hooks in a Build and Test Submit File

The following shows the submit file used to demonstrate task hooks. The task hook commands highlighted in blue. The task hook remote_task was used in the Hello World tutorial.

@project@ = tutorial
@component@ = whereAmI
@description@ = Example to demonstrate task hooks
@run_type@ = build
@inputs@ = whereAmI.cvs

The following task hooks are executed on the submit machine

{color: blue}@pre_all@ = code/perlWhereAmI/whereAmI.pl
{color: blue}@pre_all_args@ = Pre_All Task
{color: blue}@platform_pre@ = code/perlWhereAmI/whereAmI.pl
{color: blue}@platform_pre_args@ = platform_Pre Task
{color: blue}@platform_post@ = code/perlWhereAmI/whereAmI.pl
{color: blue}@platform_post_args@ = platform_Post Task
{color: blue}@post_all@ = code/perlWhereAmI/whereAmI.pl
{color: blue}@post_all_args@ = Post_All Task

The following task hooks are executed on each platform.

{color: blue}@remote_pre_declare@ = code/perlWhereAmI/whereAmI.pl
{color: blue}@remote_pre_declare_args@ = Remote_Pre_Declare Task
{color: blue}@remote_declare@ = code/perlWhereAmI/whereAmI.pl
{color: blue}@remote_declare_args@ = Remote_Declare Task
{color: blue}@remote_pre@ = code/perlWhereAmI/whereAmI.pl
{color: blue}@remote_pre_args@ = Remote_Pre Task
{color: blue}@remote_task@ = code/perlWhereAmI/whereAmI.pl
{color: blue}@remote_task_args@ = Remote_Task Task
{color: blue}@remote_post@ = code/perlWhereAmI/whereAmI.pl
{color: blue}@remote_post_args@ = Remote_Post Task
@platforms@ = x86_fc_3, sun4u_sol_5.8

Procedure

1. Review the Task Hook Diagram to understand the task hook sequence.
2. Download the attachment files into a working directory on the submit machine.

1. Open the submit file whereAmI.submit with your favorite editor and add a notify command. This will tell the build and test system how to notify you when a run is completed. Use the following format:

1. Start the submission by running nmi_submit.

Examine the Task Hook Result Files

1. Change into your run directory: /nmi/run your GID. The result files in this directory are from tasks run on the submit machine.

1. List all of the platform_pre files. As you can see there is a set of files for both of the platforms. You’ll see the results using one of several naming schemes. Again, you can ignore the nmi: at the start of the platform name for the purpose of this tutorial.

1. Each task runs a script that outputs the time it was executed. Grep for this time value in the output files.

1. Change into a platform directory to see the tasks results for a remote platform. The platform subdirectory is located at ./userdir/ platform name.

1. Run the same grep command to examine the execution times of these tasks.

1. Return to the original directory and run a combination find/grep command to print out the time of all of the tasks.

1. Examine these times. The time order should match the sequence shown in task hooks sequence shown here.

Examine Task Hooks with Build and Test Overview Page.

1. Go to the Build & Test Overview page.
2. Find the WhereAmI run.
3. Select its details page by clicking on its ID. You should see results similar to the image below.
4. Click on the ascending order button ( 1 ) in the Start column. The Name column ( 2 ) should show the task in execution order¹.
5. Notice the platform_pre and platform_post tasks. These tasks are performed once on the submit machine for every platform. The Host column shows the machine that the task was executed on. The Platform column shows which platform the tasks was executed for.

1. Click on the Output buttons ( 3 ) to see the results of each task. The whereAmI script prints out the time it was executed.

¹ The times in the Start column may be off because of clock skew problems with the machines of the Build and Test System. For example, this problem could show that the tasks run on the submit machine were executed after one or more of the platform tasks.

Procedure

1. Take the build and test submit file from the task hook tutorial, rename the whereAmI.submit file to whereAmIWithMacro.submit and add the following line to the file:

1. Submit the file to the build and test system.

Examine the Task Variables

1. The perl script (code/perlWhereAmI/whereAmI.pl) prints out all of the variables containing the characters NMI, CONDOR, and PATH. For this reason, you can look in the standard output files for each task (stored in the run directory) to determine the variables set by the build and test system.

1. Change to the user directory.

1. Search for the NMI_PLATFORM variable. As you can see it is present in every task.

1. Search for a specification file command. All of these are also present in every task.

1. Search for the _NMI_GID which is only present in the submit machine tasks. The macro is only present in log files in the run directory.

1. Search for the user macro IAmAMacro.

Examining generateOutput.pl

The script used for this tutorial is attached. The script creates a file called “CreatedFile.txt” into which each task writes and identification message. Locate and examine the following code snippet:

if (-e 'CreatedFile.txt') { open IN, 'CreatedFile.txt'; my lines = ;@ close IN; print "Found created file:\n"; for my $l (lines) { print $l;@ } } else { print "Created file not found\n"; }

This script checks for the existance of the file CreatedFile.txt in the current working directory. If the file exists, the script will read its contents and print them to standard out.

Immediately following this section is the section that appends an identification message to the file. This will also create the file if it does not exist:

open OUT, ">>CreatedFile.txt";
print OUT "Greetings from $taskhook. The time is $time.\n";
close OUT;

By tracking the contents of this file, we can see how the build and test system manages the contents of the component for the various tasks.

Procedure

generateOutput.pl Revisited

generateOutput.pl Revisited.

As was discussed previously, a results.tar.gz file needs to be created by one or more of scripts called by the remote task hooks in the build and test specification file. In our example, all of the task hooks call the same generateOutput.pl script. Locate and examine the following code snippet¹:

if (defined $genresultfile) { print "Creating results.tar.gz file\n"; system("tar czvf results.tar.gz CreatedFile.txt");
}

The $genresultfile flag that is set by the Getopt::Long module. This snippet assumes that a tar program which supports compression is visible. For this particular program this assumption is correct because the build and test system has to support the creation of the results.tar.gz. Other programs may require prerequisites which will be discussed later.

¹ A copy of the script is attached on this page.

results.tar.gz Procedure

Use the Overview Page

Use the Overview Page.

1. Open the details page for the run

1. Locate the download box. This contains links to the results.tar.gz for each platform. The image below shows where the box is located. Download one of these files and look at the CreatedFile.txt file inside.

Examining userTasks.pl

The script we will examine is attached. Rather than create a different script for each task we wish to define, we’ll use a single script, userTasks.pl, capable of performing each step, depending on its arguments.

The main execution flow of the script is contained within the hash %table. The hash contains a list of task names which are associated with a subroutine. The following snippet shows which tasks associate with what subroutines:
my %table = ( default => \&default, failure => \&failure,
timeout_failure => \&timeout_failed,
remote_declare => \&generate_tasklist_nmi,
remote_post => \&generate_results_file,
);

The table has entries for the remote_declare and remote_post tasks. The remote_declare task calls the function which creates the tasklist.nmi file. The remote_post task calls the function which generates the results.tar.gz file. It should be noted that these files can be generated during other tasks. Generating the files during these tasks follows the model that the system was designed for.

The script expects two flags, -taskhook and -fail. The remaining arguments to the script are assumed to be user defined task names that are passed as a list to the associated function in %table. The only function that uses this list is generate_tasklist_nmi.

The last thing to note is the following:

my $name = $taskhook;
$name = $ENV{'_NMI_TASKNAME'} if defined $ENV{'_NMI_TASKNAME'};

As this snippet shows, the variable $name can either contain the name of a taskhook or the name of a user defined task. The script treats taskhook tasks the same as user defined tasks.

Procedure

1. Download the attached files to the submit machine

1. Examine the file userTasks.submit. As shown in the snippet below, all of the task arguments are similar except for remote_declare. While all of the tasks are given their name with the -taskhook flag, remote_declare is also provided with user-defined task names that will be inserted into the tasklist.nmi file.

1. Run nmi_submit.

1. Open the details page for the run . In addition to the usual task names, you should see the user-defined tasks listed on the page, similar to this:


2. Click on the standard out log of one of the user tasks. Notice that the value of _NMI_TASKNAME is the same as the task name.
3. Click on the standard out log of one of the task hook tasks such as remote_post. Notice that the value of _NMI_TASKNAME is not defined.

1. Download and unpack the results file. Look at the tasklist.nmi file contained inside. It should have contents similar to this:

Add a User-Defined Task Failure: Procedure

1. Now let’s add a failed user-defined task. This can be done by simply adding the task name failure to the arguments of the remote_declare task hook. Open the file userTasks.submit into your favorite editor and change the arguments to remote_declare as shown below:

1. Run nmi_submit.

1. Open the details page for the run . You should see the user tasks listed on the page similar to this:

2. Notice the following:
   1. This is the “failure” tasked that failed.
   2. The user tasks following the failure still ran showing that this failure is of type continue remote / abort platform discussed here.
1. The failure is propagated up through the system to the user. This propagation causes the “meta-tasks” remote_task and platform_job to be marked as failed.

1. Click on the standard error log for the failure task (its the red X’d box next to the standard out button). The log should contain the statement Task failure has failed which was generated by the perl die command.

Add a Timeout Failure: Procedure

1. Now lets add a time out failure. This can be done by simply adding the task name _timeout_failure_ to the arguments of the remote_declare task hook. Open the file userTasks.submit into your favorite editor and change the arguments to remote_declare as shown below:

1. Run nmi_submit.

1. Open the details page for the run . You should see the user tasks listed on the page similar to this:

2. Notice that the time_out failure task failed as a _failure_noted_ type. This is the same behaviour as the failure task added in the previous section.

1. Click on the standard error log for the timeout_failure task and notice that the build and test system killed the task after 60 seconds.

Add a Remote Preparation Failure: Procedure

1. Now lets add a remote preparation failure. This is a failure on one of the remote tasks executed before remote_task. We are going to make this happen by adding a -fail flag to the arguments of remote_pre. Open the file userTasks.submit into your favorite editor and change the arguments to remote_pre as shown below:

1. Run nmi_submit.

1. Open the details page for the run . You should see the user tasks listed on the page similar to this:

2. Notice that the remote_prep task failed as a _remote_post / abort platform_ type (Remember failure handling). After the system detected the failure it skipped the execution of the remaining remote task and executed remote_post. After this it aborted the run.

Add a Platform-Preparation Failure: Procedure

1. Now lets add a platform-preparation failure. This is a failure of the initial platform-specific task on the submit machine. We are going to make this happen by adding a -fail flag to the arguments of platform_pre. Open the file userTasks.submit into your favorite editor and change the arguments to platform_pre as shown below:

1. Run nmi_submit.

1. Open the details page for the run . You should see the user tasks listed on the page similar to this:

2. Notice that the platform_pre task failed as a abort platform type (Remember failure handling). The platform, and thus the run as a whole (since we were running on only one platform), aborted after the failure was detected.

Procedure

1. Download the first two attached files to the submit machine. The third file is the perl script executed during the remote task. You are welcomed to look at this file to see what is being executed.

1. Run nmi_submit.

1. When the run is completed, open the details page. Notice that the run has failed.

1. Click on the error ouput for the remote_task. You should see the following:

1. The middle line shows that the script died because it could not find the md5sum program. This program is contained in the prereq coreutils.

1. Click on the details page.
2. Click on the host machine link . This will display a list of prereqs that are installed. You can check out other machines by selecting the Pool Overview link at the top of the page.

1. Locate the core utils package as shown below:


2. add a prereq requirement to the build and test specification file:

1. resubmit the file:

1. When the run is completed, open the details page. Notice that the run has passed now.

1. Click on the output for the remote_task. You should see an MD5 sum on the file TextFile.txt. Notice also that the PATH variable now contains a path to the coreutils programs. In addition, the system created a new variable called _NMI_PREREQ_coreutils_5_2_1_ROOT which contains the path to the coreutils installation. The system creates a seperate variable for every prereq requested and found as described here.

Pinning the generateOutput Run Procedure

1. Locate the run you submitted as part of the generateOutput section. Make sure that you have added the -genresult flag as part of the submission. The runid in this case is 72315.

1. Use this run id with nmi_pin to save the run.

1. Notice that this saves the run for the default 2 months. Hopefully you will be finished with this example before that time:).

Workflow Procedure

Workflow Procedure.

1. Download the attachment files into a working directory on the submit machine.
   1. Take a look at the generateMoreOutput.pl script which is executed for every task. It is similar to the generateOutput.pl script discussed here. Notice the $extras variable which is appended to the output and prepended to the output file. Notice also that the script prints the contents of any file ending with CreatedFile.txt.

1. Take a look at the file generateMoreOuput.submit a portion of which is shown below. Notice that the input command has two entries. The first generateOutput.cvs fetches the module containing the generateMoreOuput.pl script. The second generateMoreOuput.nmi to reuse a build. Notice also that the _args commands now have an extra argument MoreOuput. The script sticks this in the $extras variable discussed earlier.

1. Open the file generateMoreOutput.nmi with your favorite editor. This input file uses the nmi method which fetches the results from a previous build and unpacks into the working directory of the submit machine before platform_pre is executed. The argument we will use here is input_runids. Type in the run id you remembered from the previous section:

input_runids = 72315
2. Start the submission by running nmi_submit.

1. When the run is completed, open the details page for the run

1. Click on the standard ouput for the various tasks. The following output indicates that the tasks has found and accessed the results from the previous run:

Workflow with Multiple Runs Procedure

Workflow with Multiple Runs Procedure.

1. The nmi method’s input_runids command can accept more than one runid so lets add another run. The first thing that needs to be done is to modify the build and test specification file so that the results from its run can be differentiated from previous run. Open the file generateMoreOutput.submit with your favorite editor and replace the phrase “MoreOutput” in each *_args command to “EvenMoreOutput”[1]. For example:

1. Next add the runid from the previous workflow example to the file generateMoreOutput.nmi. Be sure the runids are seperated by a comma:

input_runids = 72315, 72330
2. Start the submission by running nmi_submit.

1. When the run is completed, open the details page.

1. Click on the standard ouput for the various tasks. The following output indicates that the tasks has found and accessed the results from both previous runs:

This output should be present in all of the tasks except pre_all and post_all.

¹ If your favorite editor is vi then you can do a global search and replace using the following command :1,$s/ MoreOutput/ EvenMoreOutput/. Be sure to preserve the space before each string otherwise you will rename the script generateEvenMoreOutput.pl which will cause the run to fail.

Workflow with Repeated Runs Procedure

Workflow with Repeated Runs Procedure.

1. Add the runid from the previous workflow example to the file generateMoreOutput.nmi. This time we are going to rerun the same submission and see what happens.

1. Start the submission by running nmi_submit.

1. When the run is completed, open the details page for the run .

1. Click on the standard output of the platform_pre task which is the first tasks that accesses the results from the previous runs. Notice this section:

1. Open generateMoreOutput.submit with your favorite editor and comment out the pre_all command and its arguments.

1. Start the submission by running nmi_submit.

1. When the run is completed, open the details page for the run .

1. Click on the standard output of the platform_pre task which is the first tasks that accesses the results from the previous runs. This time the task found the file from the previous run:

Building Zsh Manually

1. Download the source for zsh here.
2. Build it on a local machine. Follow the instructions in the INSTALL file. (Since we’re building from the source tarball, and not directly from CVS, the configure scripts are already present and you do not need run autoconf.)
3. Run the tests for zsh by typing make check. You may have some problems with some of the tests hanging. For example the Y0 test hangs on Mac OS X 10.4. The tests can also be run individually by using the TESTNUM argument. For example make TESTNUM=B check runs the B* set of tests.
4. Lets review the steps you had to do to build and test zsh:
   1. Retrieve the source code tarball from an FTP site.
   2. Unpack the source tar file.
   3. change into the source directory zsh-4.3.2.
   4. Execute ./configure. You may have had to add some flags dependending on your local machine.
   5. Execute make.

1. Execute make check.

In order to repeat this every night, or on multiple remote platforms, these steps will need to be automated. The way to do this is by adding what is commonly known as a glue script.

Simple Build and Test

Simple Build and Test.

This first cut demonstrates a minimal build and test framework that shows you how easily a traditional build process can be added into the build and test system. Download the attachments and put them into your working directory on the submit machine.

Adding User Defined Tasks

Adding User Defined Tasks.

The simple build and test run is sufficient for one-off tests. However, the real strength of the build and test system is running automated builds that are constantly repeated. Because of this, it makes sense to improve the submission files so that build and test results can be understood more easily by the casual user. In this tutorial, we will start by moving the build steps into explicitly-named user defined tasks, so that the build steps can easily be identified on the details page. To accomplish this we need to defined an NMI_TASKLIST file as we did in the user tasks tutorial.

To start this tutorial, download the attachments into your working directory on the submit machine.

Robust Build and Test Files

Robust Build and Test Files.

In the previous section we introduced the idea that the build and test system should be used to run continuously repeated tasks. This kind of use can be part of software development practices such as continuous integration. However as with other internal tools, a good design is needed up front to keep developers from falling into the trap of spending more time fixing internal tools than working on the software deliverables¹. In this tutorial we show some of the tricks that can be used to make the build and test system a vital but invisible part of your development process. We will demonstrate the following design principles:

1. Localize Anticipated Changes to the Submission Files. Some aspect of the submission files will need to be changed everytime the build and test system is used. The trick here is to anticipate what these changes are and then provide one place where these changes are made. Here are some frequent changes:
   • Input Changes. The build and test system may need to retrieve the software using a different tag, pathname, or repository.
   • Version Number. Every release of an application requires a new version number. This number can appear in a build and test run in a number of places. For example, the version number for the ZShell example shows up in the name of the archive and in the path to the working directory of the build.
   • Test Customization. The purpose of a test and build run is to run tests that detect errors. A run that fails needs to be re-run with different parameters in order to isolate and debug the problem that caused the failure.
• Platform changes. The list of platforms that constantly modified based on testing needs and the availability of platforms.
• Provide Readable Logs. A build and test run is executed as a batch job. Because of this, the only way to understand the cause of failures is to scan the output and error logs. From these logs the failure has to be isolated and the information to reproduce the failure needs to be collected. One of the things we will show here is to provide information about each step that is executed such as:
• The full path of the executable.
• The command line arguments.

• The working directory that the command is executed in.

We will use four features of the build and test system.

• The user defined tasks introduced in the previous section.
• Build and test system variables and user macros which were demonstrated here.

• Build and test system shell variable substitution explained here.

¹ This trap is humorously explained in the book The Peter Pyramid. In the book, Dr. Peter explains that one of the symptoms of an organization suffering from to much bureaucracy is when it spends more time on external processes then in talking to its customers.