This document details the configuration and use of the 3G Bridge Metajob feature.
The Metajob feature of the 3G Bridge enables the user to submit a batch of jobs exactly as they would a single job. Because of this transparency, the Metajob feature can be used even when jobs are submitted to the Bridge through the gLite infrastructure.
Formally, a Metajob is a 3G Bridge job with and extra input file—called the
Metajob file— with the name prefix '_3gb-metajob' . This extra input
file contains the definition of the sub-jobs of the Metajob.
The Metajob itself is used as a template for its sub-jobs. The Metajob file contains instructions that modify the current template, and instructions to create sub-jobs from the current template.
To submit a Metajob, you must create the Metajob definition first, and then, submit it to the 3G Bridge. As the information specified in the submission is used as the initial template for the sub-jobs, the example submission is presented first:
wsclient -m add -e 'http://example.com:8091/' \ -g test -n app \ -i _3gb-metajob-example=http://example.com/inputs/_3gb-metajob-example \ -i alpha.txt=http://example.com/inputs/alpha_1.txt \ -i beta.txt=http://example.com/inputs/beta_1.txt \ -a '--p=1 --in1=alpha.txt --in2=beta.txt' -o result.txt -o stats.txt
Notice that aside from the extra input file '_3gb-metajob-example' , this
submission is just an ordinary job submission. All attributes, including the
target queue (grid+algorithm name), is the same as if you'd submit a single
job. This submission—excluding the Metajob definition file—will be used as
the initial template. The current state of the template can be changed in the
Metajob definition:
_3gb-metajob-example: ... Input=alpha.txt=http://example.com/inputs/alpha_1.txt=76ebd8e3bcb00ca8a47db97cabfc6f15=1536 Input=beta.txt=http://example.com/inputs/beta_1.txt=6e70ca01869f23676840f7d9fb51312a=9586 Arguments=--p=2 --in1=alpha.txt --in2=beta.txt ...
These commands can be defined multiple times; all occurrence will change the
current template, overwriting its previous state. When the current template
describes a sub-job we want to submit, a sub-job—or several identical
sub-jobs—can be instantiated with the Queue command:
Queue
or, for example,
Queue 10
The Note that:
'Input=alpha.txt' command sets the location of alpha.txt in the current template (same for beta.txt). No new input files can be defined and none of them can be removed from the template. This implies that
As the Metajob is itself a 3G Bridge job, it has a status attribute, which must
must be determined based on the state of its sub-jobs. The trivial case is when
all sub-jobs have finished successfully; in this case, the status of the Metajob
can be FINISHED. It is also trivial, when all sub-jobs have failed. But what
happens, when some of the jobs have successfully finished, while others have
failed? The 3G Bridge allows the user to control its behaviour in these
intermediary cases. In the Metajob file, the user can specify the minimum and
maximum number of sub-jobs they need to successfully finish.
The lower limit tells the Bridge that the whole Metajob has to be considered failed, and no output is produced if less than this number of sub-jobs have successfully finished. If the number of failed sub-jobs reaches the point where the lower limit becomes impossible to reach, the Bridge prematurely cancels all pending sub-jobs, and the Metajob fails immediately. This is useful when the result of the batch is useful, and sub-results are not needed. If this limit is set to 1, any successful sub-result will be available after the Metajob has finished.
The upper limit tells the Bridge that no more than this number of sub-results is needed. If the number of successfully finished sub-jobs reaches this number, the Bridge cancels all pending sub-jobs, and the Metajob finishes immediately. This is useful to introduce redundancy at user level, if the results are interchangeable (Monte Carlo simulations for example).
These limits can be specified in the Metajob file; both %Minimum and
%Maximum can be specified once. This example shows how the limits can be
specified. The term All and percentages refer to the number of
sub-jobs defined in the Metajob file.
Default value for both limits is All.
%Minimum 5 %Minimum 20% %Minimum All %Maximum 50 %Maximum 80% %Maximum All
The following example shows a correct Metajob submission. The first Queue
will create 100 instances of the initial template matching the submission
information. The second 100 sub-jobs will have the source of alpha.txt
changed, all other attributes being unchanged; etc.
wsclient -m add -e 'http://example.com:8091/' \ -g test -n app \ -i _3gb-metajob-example=http://example.com/inputs/_3gb-metajob-example \ -i alpha.txt=http://example.com/inputs/alpha_1.txt \ -i beta.txt=http://example.com/inputs/beta_1.txt \ -a '--p=1 --in1=alpha.txt --in2=beta.txt' -o result.txt -o stats.txt
_3gb-metajob-example: %Minimum 1 %Maximum 80% Queue 100 Input=alpha.txt=http://example.com/inputs/alpha_2.txt=76ebd8e3bcb00ca8a47db97cabfc6f15=1536 Queue 100 Input=beta.txt=http://example.com/inputs/beta_2.txt=6e70ca01869f23676840f7d9fb51312a=9586 Queue 100 Arguments=--p=2 --in1=alpha.txt --in2=beta.txt Queue 100
The status of the submitted Metajob can be queried the same way as of a normal
job (wsclient -m status). However, the status of a Metajob is only an
aggregation of the statuses of its sub-jobs. Detailed information about the
sub-jobs is provided through a URL, which is stored in the Metajob's
griddata attribute:
wsclient -m griddata -e 'http://example.com:8091/' -j <MetaJobID>
The output files produced by sub-jobs cannot be accessed separately. Instead,
when the Metajob has successfully finished, an archive containing the output of
successful sub-jobs is created for each logical output file. In our running
example, two output files will be created, result.txt and stats.txt;
both being a .tar.gz archive (without the extension) containing respective
output files of the sub-jobs.
wsclient -m griddata -e 'http://example.com:8091/' -j <MetaJobID>
Output:
<MetaJobID> result.txt http://example.com/download/<...>/result.txt <MetaJobID> stats.txt http://example.com/download/<...>/stats.txt
To obtain the results, download and extract them:
wget http://example.com/download/<...>/result.txt wget http://example.com/download/<...>/stats.txt tar xzf result.txt tar xzf stats.txt
The result of these commands will be a directory structure, in which directory
names are UUID-s (those of the sub-jobs), organized in higher level directories
(first two characters of the UUID). In each directory, there will be the
corresponding result.txt and stats.txt.
tree
.
├── 0b/
│ └── 0b3e6bd3-f8b4-4d60-84a9-8c9c5855ffce/
│ ├── result.txt
│ └── stats.txt
├── 26/
│ └── 263a6140-cf01-4889-ad21-210ecd3d41c4/
│ ├── result.txt
│ └── stats.txt
├── 47/
│ ├── 475ea5d0-982d-45d2-8260-7eb8493851e5/
│ │ ├── result.txt
│ │ └── stats.txt
│ └── 4787ab3d-594f-4691-96db-b826c3d63b61/
│ ├── result.txt
│ └── stats.txt
└── 82/
└── 826640ec-f67a-4a1f-a937-35edcd00d4b3/
├── result.txt
└── stats.txt
The Metajob feature is implemented as a plugin for the 3G Bridge. To enable the Metajob feature
The name of the queue must be 'Metajob' (case-sensitive).
mysql boinc_<projectname> << EOF INSERT INTO cg_algqueue (grid, alg, batchsize) VALUES ('Metajob', 'algname_1', 10); INSERT INTO cg_algqueue (grid, alg, batchsize) VALUES ('Metajob', 'algname_2', 10); INSERT INTO cg_algqueue (grid, alg, batchsize) VALUES ('Metajob', 'algname_3', 10); ... EOF
3g-brigde.conf: ... [Metajob] handler = Metajob maxJobsAtOnce = 100
The only configuration option is maxJobsAtOnce which determines the
number of sub-jobs to unfold in one run. After unfolding this many jobs, the
state is saved, and the control is given back to the
QueueManager. In the next iteration, the unfolding
continues from the saved state. Higher values allow faster unfolding of
individual Metajobs, but may impair the overall throughput of the QueueManager.
Don't forget to configure the DownloadManager plugin too, if you need the Metajob functionality.
