Job-level accounting information interface

Not relased yet. [draft]

The 3G Bridge accounting interface provides information that is required by the EGI Grid Federation to accept Desktop Grids as federation components.

Schema specification

The interface provides accounting information on a job-by-job basis. The information available through this interface is described in the following table.

Name Unit Description
start_time UNIX timestamp Start time of the job as reported by the DG
end_time UNIX timestamp End time of the job as reported by the DG
wallclock_time s Start time - End time
cpu_time s Net CPU time consumed by the job
memory byte Approx. memory consumed by the job (see remarks below)
host_flops 1/s Floating point performance of the executing host
host_intops 1/s Integer performance of the executing host
host_ncpus 1 Number of processors in the executing host
Remarks
  1. The values are extracted from the desktop grid database.
  2. The desktop grid executes jobs redundantly; only the canonical result is used to determine the accounting information.
  3. The memory consumed by the application cannot be determined exactly. The memory bound defined for the application is used as an approximation.
  4. This information is only available so long as the job exists in the database. After deleting the job, accounting information is lost.

Dealing with meta-jobs

The following are implementational details included for the sake of completeness; the client using this interface need not be concerned about it.

Meta-jobs present two issues.

First, the accounting information of a meta-job must be an aggregation of information on its sub-jobs. The host_flops and host_intops values are particularly problematic. For example, the host_flops of the participating hosts cannot simply be summed: each term must be weighted with the actual participation of the given host. The solution is the following. The number of floating point operations performed on each host is computed. The total number of operations divided by the total CPU time will give the actual performance of the infrastructure executing the meta-job:

aggregate_host_flops = sum(cpu_time*host_flop)/sum(cpu_time)

The aggregation function used for meta-jobs are summarized in the following table:

Name Aggregator in case of meta-jobs
start_time Min
end_time Max
wallclock_time Max - Min
cpu_time Sum
memory Sum
host_flops Sum(cpu_time*host_flops)/Sum(cpu_time)
host_intops Sum(cpu_time*host_intops)/Sum(cpu_time)
host_ncpus Sum

Second, as a meta-job progresses, when a sub-job has finished, its output is processed and the sub-job is deleted immediately. Therefore, the sub-job information must be saved before it is lost. This is done in database internally, with a trigger. The saved information is aggregated to produce the accounting information of the meta-job.

Enabling the interface

The interface is provided only by the REST interface. The interface is enabled by default. To disable it, remove the corresponding line from 3gb.php:

3gb.php
RESTHandler::addHandler('AccinfoHandler');
...
RESTHandler::addHandler('JobsHandler');

Make sure that the registration of AccinfoHandler precedes the registration of JobsHandler, as the URL patterns of these handlers overlap.

The database has to provide the necessary functionality to enable the interface. The schema is defined in db/egi-accounting.sql.

mysql << db/egi-accounting.sql

Interface specification

The accounting information can be accessed through <job's URL>/accinfo

For example:

JOBID=e8d8e4f4-27f0-409f-ba80-f9f9e33d3242
wget "http://home.edges-grid.eu/home/download/wss/jobs/$JOBID/accinfo"
e8d8e4f4-27f0-409f-ba80-f9f9e33d3242 1361961911 1362010369 48458 1259713.9488 1339227204000 27077851019757.2 82720014025364.8 47083

The result is the record containing the job identifier followed by the fields described in the table above in order. The syntax is as described in the REST interface manual (space delimited lines by default).

manual/restif-accounting.txt · Last modified: 2013/03/11 14:17 by a.visegradi
Trace: restif-accounting
Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0