User Tools

Site Tools


manual:restif

REST interface for the 3G Bridge

This manual describes the use and configuration of the 3G Bridge REST interface.

User Guide

The REST interface is an alternative to the SOAP based interface, with similar functionalities. The only limitation is that local files cannot be specified as input files when submitting a job; only remote file specifications are possible. Aside from this, the REST interface is a superset of the SOAP interface.

In this manual, we assume that the project URL is http://example.com/myproject/, and the REST interface is deployed in download/wss/. That is, the base URL of the REST interface is http://example.com/myproject/download/wss/.

General options

A requested entity or collection will respond with a table (list of associative arrays). This response will be rendered according to the format argument, which can be plain, html or json. The hdr argument enables (1) or disables (0) the rendering of a header if applicable. If the Content-Type of the request is 'application/json' , the response will be in JSON format. Otherwise, the default response format is plain.

  • plain: The response will be rendered as plain text. Records are separated with '\n', attributes are separated with a single space. Separators can be overridden with the sep and rsep (record sepatator) arguments. A header is rendered if requested (hdr=1).
  • html: The response is rendered as a HTML table. A header is rendered by default.
  • json: The response is rendered as a JSON string.

Example:

http://example.com/myproject/download/wss/version
3G Bridge 1.9
http://example.com/myproject/download/wss/version?hdr=1
version
3G Bridge 1.9
http://example.com/myproject/download/wss/version?format=json
[{"version":"3G Bridge 1.9"}]

For plain format, the field separator can be overridden with the sep argument. The record separator can be overridden with rsep.

http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d?sep=|
093fd32b-4604-4c06-9ba9-d3aae7a6b80d|Cloud|cloud|CANCEL||--image=ami-00000021|||2012-09-11 11:20:06|

For most entities (not collections), specific attributes can be selected in the URL:

http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d/id+status+creation_time
93fd32b-4604-4c06-9ba9-d3aae7a6b80d CANCEL 2012-09-11 11:20:06

Entities

Most entities only support the GET verb. If not stated otherwise, the following output examples are the results of GET requests on the specified entities.

Querying the version of the Bridge

http://example.com/myproject/download/wss/version
3G Bridge 1.9

SOAP equivalent:

wsclient -m version -e http://example.com:8091/

Querying queues

Querying available queues
http://example.com/myproject/download/wss/queues
Cloud audiveris 1
edgidemo audiveris 1 
edgidemo bwa-boinc 1
NULL dsp
MetaJob dsp
...

SOAP equivalent: —

Querying available grids
http://example.com/myproject/download/wss/grids
Cloud
edgidemo
MetaJob
NULL

SOAP equivalent: —

Querying available algorithms for a grid
  1. All attributes:
    http://example.com/myproject/download/wss/grids/edgidemo
  2. Only algorithm names:
    http://example.com/myproject/download/wss/grids/edgidemo/alg
edgidemo audiveris 1 
edgidemo bbgc 1 
edgidemo Blender 1 
edgidemo bwa-boinc 1
...

SOAP equivalent for 2nd case:

wsclient -m gridalgs -e http://example.com:8091/ -g edgidemo

Managing jobs

Querying all jobs
http://example.com/myproject/download/wss/jobs?hdr=1
id grid alg status gridid args griddata tag creation_time metajobid
093fd32b-4604-4c06-9ba9-d3aae7a6b80d Cloud cloud CANCEL  --image=ami-00000021   2012-09-11 11:20:06 
0bba95f9-38ea-4cf9-8125-6bc3e93af154 Cloud cloud CANCEL  --image=ami-00000021   2012-09-13 10:40:07
...

SOAP equivalent: —

Querying a single job
http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d
093fd32b-4604-4c06-9ba9-d3aae7a6b80d Cloud cloud CANCEL  --image=ami-00000021   2012-09-11 11:20:06

Any subset of the attributes can be selected. Consider the following SOAP commands:

wsclient -m status -e http://example.com:8091/ -j 093fd32b-4604-4c06-9ba9-d3aae7a6b80d

REST equivalent:

http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d/id+status
093fd32b-4604-4c06-9ba9-d3aae7a6b80d CANCEL
wsclient -m griddata -e http://example.com:8091/ -j 093fd32b-4604-4c06-9ba9-d3aae7a6b80d

REST equivalent:

http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d/id+griddata
093fd32b-4604-4c06-9ba9-d3aae7a6b80d d2048dd6-464e-47c9-a4d0-373a9de25bdb_e933a6f8-7365-4b58-846d-a38be7d4ba24_76a26b38-88fd-4ba3-912b-2b1e30ed74d6
Querying finished jobs
http://example.com/myproject/download/wss/jobs/finished/id+status
093fd32b-4604-4c06-9ba9-d3aae7a6b80d FINISHED
0bba95f9-38ea-4cf9-8125-6bc3e93af154 ERROR
0f25e8cb-45f6-4f7f-83b4-4525dc245b11 FINISHED

SOAP equivalent:

wsclient -m finished -e http://example.com:8091/
Cancelling a job or deleting a finished job

To cancel a job, send a request to its URL with the DELETE verb.

curl -X DELETE  http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d

SOAP equivalent:

wsclient -m delete -e http://example.com:8091/ -j 093fd32b-4604-4c06-9ba9-d3aae7a6b80d
Submitting a job

A job can be submitted to the Bridge by POSTing a job object to the jobs entity. Currently, there are two ways to do this; either by posting form-data (Content-Type: application/x-www-form-urlencoded) or by posting a JSON object (application/json). The following examples show how this can be achieved using curl.

In both cases, by default, after a successful submission, the Bridge will respond with a redirect header to the newly created jobs (Location: http://example.com/myproject/download/wss/jobs/<new_uuid>). The -L switch tells curl to follow this redirection.

Another possibility is to specify 'redir=0' in the URL (…/wss/jobs?redir=0); in this case, no redirection occurs, only the unique id of the new job is printed. This is the same behaviour as that of the SOAP interface.

To submit a job, the same information has to be supplied as in case of the SOAP interface. Notice that—unlike the SOAP interface—the URL, MD5 and size parts of a specification are specified separately, not in a single string joined with '=' .

Submitting using form data
Submitting a job using CURL
curl \
    -d name=dsp \
    -d grid=NULL \
    -d args='-dpf=docking.dpf' \
    -d tag='Test job' \
    -d input[docking.dpf][url]=http://example.com/myproject/download/inputs/publicautodock423/docking.dpf \
    -d input[docking.dpf][md5]=a7bac5a941a85660cd87bf83f93b852a \
    -d input[docking.dpf][size]=2933 \
    -d input[inputs.zip][url]=http://example.com/myproject/download/inputs/publicautodock423/inputs.zip \
    -d input[inputs.zip][md5]=cbb8c0ada8e197d6db707e6ce4f03ca6 \
    -d input[inputs.zip][size]=3270896 \
    -d output[0]=log.dlg \
    http://example.com/myproject/download/wss/jobs?hdr=1 -L
id grid alg status gridid args griddata tag creation_time metajobid userid error_info
022a9b75-8a08-4f5d-8188-c8687659e60d NULL dsp INIT  -dpf=docking.dpf  Test job 2013-01-22 13:30:32   

Alternatively:

Submitting a job using CURL
curl \
    # [ same as above ] \
    http://example.com/myproject/download/wss/jobs?redir=0
022a9b75-8a08-4f5d-8188-c8687659e60d
Submitting using JSON

This is an example job description in JSON:

example-job.json
[
    {
        "name": "dsp",
        "grid": "NULL",
        "args": "-dpf=docking.dpf",
        "tag": "Test job",
        "input": {
            "docking.dpf": {
                "url": "http://example.com/myproject/download/inputs/publicautodock423/docking.dpf",
                "md5": "a7bac5a941a85660cd87bf83f93b852a",
                "size": 2933
            },
            "inputs.zip": {
                "url": "http://example.com/myproject/download/inputs/publicautodock423/inputs.zip",
                "md5": "cbb8c0ada8e197d6db707e6ce4f03ca6",
                "size": 3270896
            }
        },
        "output": [
            "log.dlg"
        ]
    }
]

This example job can be submitted with the following command.

Submitting a JSON job
curl \
    --header 'Content-Type: application/json' \
    --data @example-job.json \
    http://example.com/myproject/download/wss/jobs
[{"id":"98cba344-80cd-451a-bca8-cd3c887d0731"}]
If plain text output is required
curl \
    --header 'Content-Type: application/json' \
    --data @example-job.json \
    http://example.com/myproject/download/wss/jobs?format=plain
af3b0de8-c2cc-43eb-ba27-67ae1cda27db

Admin Guide

Follow these steps to deploy the REST interface for the Bridge:

  1. Create a directory that will be published by the project web server. For example, if you create a directory named wss in project/download, the directory will be available through http://<projecturl>/download/wss.
  2. Setup the project site to allow RewriteEngine and security overrides in .htaccess files.
  3. Allow write access to the Bridge output directory.
  4. Setup security.
  5. Copy the php scripts that constitute the interface to the directory just created.
  6. Configure the interface by setting the necessary values in 3gb.php.
Create the directory
mkdir $HOME/project/download/wss
Setup Apache

Setup Apache and the project site to support the Rewrite module and security overrides.

Enabling the Rewrite engine as ''root''
sudo a2enmod rewrite

In the project site configuration ([project_home]/project/[project_name].httpd.conf), FileInfo and Limit has to be added to AllowOverride for directory …/user. This enables the use of RewriteEngine and security overrides in .htaccess.

As output directories will be created upon submission, PHP must be allowed to access the 3G Bridge output directory (as configured in 3gbridge.conf/[wssubmitter]/output-dir). To do so, add this directory, delimited with a colon, to the open_basedir PHP admin value.

[project_home]/project/[project_name].httpd.conf
[...]
<Directory "[project_home]/project/html/user">
   [...]
    AllowOverride AuthConfig FileInfo Limit
   [...]
    php_admin_value "open_basedir"    "...:[project_home]/3g-bridge/output"
[...]   

Restart Apache for these changes to take effect:

sudo service apache2 restart
Allow write access to the Bridge output directory

Assuming the Bridge output directory is [project_home]/master/3g-bridge/output. The actual value can be found in the 3G Bridge configuration file, in the [wssubmitter] section.

Make sure that the Apache process user (usually www-data) is added to the project user's default group

getent group $USER
boinc-home:x:108:www-data

If www-data is not listed next to the project user default group, you must correct this:

sudo usermod -a -G $USER www-data

When done, add group write access to the output directory and its direct descendants:

Make the output directories writable to the Apache user
cd $HOME/master/3g-bridge
chmod 775 `find output -maxdepth 1 -type d`

Create a logfile, which is writable by the Apache user:

Create a log file writable by the Apache user
LOGFILE="$HOME/project/log_<dir>/restinterface.log"
touch "$LOGFILE"
chmod g+w "$LOGFILE"
Configure Apache for the REST interface

The following is an example .htaccess file we use, which enables the use of “pretty” URLs, and restricts access to our internal network. You can setup any kind of security Apache supports.

[project_home]/project/download/wss/.htaccess
Order deny,allow
Deny from all
Allow from lpds.sztaki.hu
 
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /[project_name]/download/wss/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ 3gb.php/$1 [L]
</IfModule>
Configure the interface

To configure the interface, set the following values in config.inc.php. Usually, this involves only the following steps: (1) create a hard link to the Bridge configuration in the wss directory, (2) set the base URL of the REST interface, and (3) set the path of the log file. For completeness, other possibilities are also detailed here.

Set the base URL of the web service. It is required for redirects.

Set the path of the log file. This file must be writable by the Apache user.

Set the log level. Only messages with level at least this high will be logged. Possible settings—in order, highest to lowest—are:

  1. CRITICAL – unrecoverable errors, unhandled exceptions.
  2. ERROR – handled, but notable errors.
  3. AUDIT – produces exactly two entries for each request:
    1. For security reasons, the fact that a request is being handled, including information on the client.
    2. A single line summarising what happened.
  4. INFO – Detailed information on what happened.
  5. DEBUG – Debug information useful for developers.
  6. TRACE – Debug information useful for nobody.

For security reasons, we advise you to never set the level higher than AUDIT. Lower settings may produce extremely large amount of log data.

The CONFIG_FILE constant must point to the 3G Bridge config file. If Apache does not allow to follow symlinks or to access files anywhere on the filesystem, create a hard link to it in the wss dir.

Create a hard link to the 3G Bridge config file
cd [project_home]/project/download/wss
ln [project_home]/master/3g-bridge/3g-bridge.conf

The Bridge version is queried by executing BRIDGE_VER_CMD. Make sure the path it uses points to the 3g-bridge binary. The default setting is usually right.

Set the destination grid for meta-jobs (the name of the grid defined in 3g-bridge.conf). The default value is usually right.

Example: [project_home]/project/download/wss/config.inc.php
define('BASE_URL', 'https://PROJECT_URL/download/wss');
define('LOG_FILE', '/var/lib/boinc/home/project/log_home/restif.log');
define('LOG_LEVEL', 'AUDIT');
define('CONFIG_FILE', '3g-bridge.conf');
define('BRIDGE_VER_CMD',
       '/bin/bash -c /usr/sbin/3g-bridge\ -V');
define('METAJOB_GRID', 'Metajob');
manual/restif.txt · Last modified: 2013/05/13 12:16 (external edit)