This document details the configuration and use of the 3G Bridge Metajob feature.

The Metajob feature of the 3G Bridge enables the user to submit a batch of jobs exactly as they would a single job. Because of this transparency, the Metajob feature can be used even when jobs are submitted to the Bridge through the gLite infrastructure.

Formally, a Metajob is a 3G Bridge job with and extra input file—called the Metajob file— with the name prefix '_3gb-metajob' . This extra input file contains the definition of the sub-jobs of the Metajob.

The Metajob itself is used as a template for its sub-jobs. The Metajob file contains instructions that modify the current template, and instructions to create sub-jobs from the current template.

To submit a Metajob, you must create the Metajob definition first, and then, submit it to the 3G Bridge. As the information specified in the submission is used as the initial template for the sub-jobs, the example submission is presented first:

wsclient -m add -e 'http://example.com:8091/' \
  -g test -n app \
  -i _3gb-metajob-example=http://example.com/inputs/_3gb-metajob-example \
  -i alpha.txt=http://example.com/inputs/alpha_1.txt \
  -i beta.txt=http://example.com/inputs/beta_1.txt \
  -a '--p=1 --in1=alpha.txt --in2=beta.txt'
  -o result.txt
  -o stats.txt

Notice that aside from the extra input file '_3gb-metajob-example' , this submission is just an ordinary job submission. All attributes, including the target queue (grid+algorithm name), is the same as if you'd submit a single job. This submission—excluding the Metajob definition file—will be used as the initial template. The current state of the template can be changed in the Metajob definition:

_3gb-metajob-example:

...
Input=alpha.txt=http://example.com/inputs/alpha_1.txt=76ebd8e3bcb00ca8a47db97cabfc6f15=1536
Input=beta.txt=http://example.com/inputs/beta_1.txt=6e70ca01869f23676840f7d9fb51312a=9586
Arguments=--p=2 --in1=alpha.txt --in2=beta.txt
...

These commands can be defined multiple times; all occurrence will change the current template, overwriting its previous state. When the current template describes a sub-job we want to submit, a sub-job—or several identical sub-jobs—can be instantiated with the Queue command:

Queue

or, for example,

Queue 10

The Note that:

1. For the input files, only their location can be redefined. The 'Input=alpha.txt' command sets the location of alpha.txt in the current template (same for beta.txt). No new input files can be define
   manual/3gbplugin-metajob.txt · Last modified: 2013/01/18 09:22 by a.visegradi