This document details the configuration and use of the 3G Bridge Metajob feature.
The Metajob feature of the 3G Bridge enables the user to submit a batch of jobs exactly as they would a single job. Because of this transparency, the Metajob feature can be used even when jobs are submitted to the Bridge through the gLite infrastructure.
Formally, a Metajob is a 3G Bridge job with and extra input file—called the
Metajob file— with the name prefix '_3gb-metajob'
. This extra input
file contains the definition of the sub-jobs of the Metajob.
The Metajob itself is used as a template for its sub-jobs. The Metajob file contains instructions that modify the current template, and instructions to create sub-jobs from the current template.
To submit a Metajob, you must create the Metajob definition first, and then, submit it to the 3G Bridge. As the information specified in the submission is used as the initial template for the sub-jobs, the example submission is presented first:
wsclient -m add -e 'http://example.com:8091/' \ -g test -n app \ -i _3gb-metajob-example=http://example.com/inputs/_3gb-metajob-example \ -i alpha.txt=http://example.com/inputs/alpha_1.txt \ -i beta.txt=http://example.com/inputs/beta_1.txt \ -a '--p=1 --in1=alpha.txt --in2=beta.txt' -o result.txt -o stats.txt
Notice that aside from the extra input file '_3gb-metajob-example'
, this
submission is just an ordinary job submission. All attributes, including the
target queue (grid+algorithm name), is the same as if you'd submit a single
job. This submission—excluding the Metajob definition file—will be used as
the initial template. The current state of the template can be changed in the
Metajob definition:
_3gb-metajob-example: ... Input=alpha.txt=http://example.com/inputs/alpha_1.txt=76ebd8e3bcb00ca8a47db97cabfc6f15=1536 Input=beta.txt=http://example.com/inputs/beta_1.txt=6e70ca01869f23676840f7d9fb51312a=9586 Arguments=--p=2 --in1=alpha.txt --in2=beta.txt ...
These commands can be defined multiple times; all occurrence will change the
current template, overwriting its previous state. When the current template
describes a sub-job we want to submit, a sub-job—or several identical
sub-jobs—can be instantiated with the Queue
command:
Queue
or, for example,
Queue 10
The Note that:
'Input=alpha.txt'
command sets the location of alpha.txt
in the current template (same for beta.txt
). No new input files can be define