Cross-site migration

Cross-site migration improvements ( now -> month 3 )

Purpose

Get automatic migration working, reliably, in production, so ETICS users’ jobs will automatically run remotely on exotic platforms they do not have in their local pool. (Remember that for now, the purpose is not to allow users to run remotely in order to get extra cycles on platforms that they do have in-house.)

Target platforms (as of ?when?, according to ?who?) include

• x86_cent_4.3
• macos_10.4
• x86 & x86_64 debian_4
• x86_rhap_5
• x86_rhas_4

• x86_RHclones_4

Plan

Right now we’re stuck at step 0, and need to demonstrate that the core migration architecture is stable.

The automated testing shouldn’t involve gLite jobs at all, at least for now. The first step is to verify that the software we’re providing (Condor + Metronome) works. If other applications are making that difficult, we need to cut them out of our testing process until our own tests are 100% clean.

Once our software stack is “clean” (see below) then we can more confidently go back and suggest that any problems may be with gLite. But I wouldn’t be comfortable doing that until we can demonstrate that our software works reliably (via exerciser results).

That’s the approach to use in general. Don’t add anything to the exerciser mix until the simple things work 100%. Then add one variable or software stack component at a time until it works 100%. Repeat until we’re replicating something as close as possible to a production ETICS job on the production ETICS infrastructure. But don’t start there.

If this necessitates two testbeds, so be it. A “testbed” is really nothing more than a distinct Condor/Metronome configuration on each end, so having more than one of them should not be a big problem (even if you only end up being able to get one machine on each end).

As for what “clean” or “failed” means, anything other than the job successfully running to completion and having its results sent back whole within a predefined reasonable amount of time (which will depend on the test — hello world vs ETICS jobs, dedicated CPU slot vs. not, etc.) should be considered a failure.

But the whole point is to start understanding and categorizing the types of failures and their frequency. The desired outcome is a report like:

Over the past month, 23% of jobs (1000) failed, with a min/median/max of 0%/7%/100% per night.
Of all failed jobs:
52% (520) timed out due to matchmaking failures
20% (200) timed out due to ETICS Condor pool configuration bugs
12% (120) failed due to NMI Lab machine failures
8% (80) timed out due to unavailable resources
4% (40) failed to due ETICS network configuration bugs
4% (40) are unknown.

Tasks

All tasks are Becky’s unless indicated otherwise.

Current

• identify what we currently believe is the “best” combination of Condor and Metronome versions for stable job migration
• this is a smaller and distinct task from fully-populating a compatibility matrix — you can create one, but just fill in what we do know, and leave what we don’t know blank for now
• identify what submit host at each site will be used for the initial testbed — ideally both should be dedicated to the exerciser for now

• set up the testbed: install the aforementioned software on the aforementioned machines (where schedd A=etics and schedd B=batlab)
the exerciser is currently running from lxb2180.cern.ch -> nmi-etics.cs.wisc.edu — is lxb2180 dedicated to the exerciser?
• confirm that the testbed supports migration by manually submitting a trivial (e.g., hello world) job from A to B, for a single platform, to a dedicated exerciser CPU slot that will always run
• develop an exerciser to automatically submit jobs like the one above, and to track and email their success/failure
• check the exerciser, emailer, and any/all support files into CVS

• Send daily and weekly reports to show results of migrated jobs. Reports should clearly indicate whether migration is working or not and identify failures. reports are sent daily and weekly to nmi-staff list and Marian.
  • Expand list to include all of SA1, Miron?

• identify and categorize every job failure until the testbed is 100% working, or the remaining failures are 100% understood and not critical, or not due to the Condor/Metronome software stack

The last step might take a while. :)

Future

Then, in some order, we can:

• run the exerciser from the production ETICS schedd rather than the artificial testbed schedd
• make the trivial job into a realistic ETICS job.
• run the job on non-dedicated CPU slots (to test the matchmaking infrastructure and policy)
• run the job on more than one platform
• experiment with advanced policy and configuration on the testbed machines. Document known good (and bad) policies and concerns.

• etc.

and

• improve exerciser, nmi_resource_advertiser, and other related Metronome infrastructure to make job migration simpler to implement and debug

We can discuss which if any of these additional steps should be done on top of the initial testbed infrastructure or done in parallel with it as separate, new exerciser instances.