User Tools

Site Tools


Sidebar

Overview

Navigation

Scenarios

Components

Manuals

  • Basic components
  • Application support
  • Additional components
    • Command disabled: revisions
      manual:restif

      REST interface for the 3G Bridge

      This manual describes the use and configuration of the 3G Bridge REST interface.

      User Guide

      The REST interface is an alternative to the SOAP based interface, with similar functionalities. The only limitation is that local files cannot be specified as input files when submitting a job; only remote file specifications are possible. Aside from this, the REST interface is a superset of the SOAP interface.

      In this manual, we assume that the project URL is http://example.com/myproject/, and the REST interface is deployed in download/wss/. That is, the base URL of the REST interface is http://example.com/myproject/download/wss/.

      General options

      A requested entity or collection will respond with a table (list of associative arrays). This response will be rendered according to the format argument, which can be plain, html or json. The hdr argument enables (1) or disables (0) the rendering of a header if applicable. If the Content-Type of the request is 'application/json' , the response will be in JSON format. Otherwise, the default response format is plain.

      • plain: The response will be rendered as plain text. Records are separated with '\n', attributes are separated with a single space. Separators can be overridden with the sep and rsep (record sepatator) arguments. A header is rendered if requested (hdr=1).
      • html: The response is rendered as a HTML table. A header is rendered by default.
      • json: The response is rendered as a JSON string.

      Example:

      http://example.com/myproject/download/wss/version
      3G Bridge 1.9
      http://example.com/myproject/download/wss/version?hdr=1
      version
      3G Bridge 1.9
      http://example.com/myproject/download/wss/version?format=json
      [{"version":"3G Bridge 1.9"}]

      For plain format, the field separator can be overridden with the sep argument. The record separator can be overridden with rsep.

      http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d?sep=|
      093fd32b-4604-4c06-9ba9-d3aae7a6b80d|Cloud|cloud|CANCEL||--image=ami-00000021|||2012-09-11 11:20:06|

      For most entities (not collections), specific attributes can be selected in the URL:

      http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d/id+status+creation_time
      93fd32b-4604-4c06-9ba9-d3aae7a6b80d CANCEL 2012-09-11 11:20:06

      Entities

      Most entities only support the GET verb. If not stated otherwise, the following output examples are the results of GET requests on the specified entities.

      Querying the version of the Bridge

      http://example.com/myproject/download/wss/version
      3G Bridge 1.9

      SOAP equivalent:

      wsclient -m version -e http://example.com:8091/

      Querying queues

      Querying available queues
      http://example.com/myproject/download/wss/queues
      Cloud audiveris 1
      edgidemo audiveris 1 
      edgidemo bwa-boinc 1
      NULL dsp
      MetaJob dsp
      ...

      SOAP equivalent: —

      Querying available grids
      http://example.com/myproject/download/wss/grids
      Cloud
      edgidemo
      MetaJob
      NULL

      SOAP equivalent: —

      Querying available algorithms for a grid
      1. All attributes:
        http://example.com/myproject/download/wss/grids/edgidemo
      2. Only algorithm names:
        http://example.com/myproject/download/wss/grids/edgidemo/alg
      edgidemo audiveris 1 
      edgidemo bbgc 1 
      edgidemo Blender 1 
      edgidemo bwa-boinc 1
      ...

      SOAP equivalent for 2nd case:

      wsclient -m gridalgs -e http://example.com:8091/ -g edgidemo

      Managing jobs

      Querying all jobs
      http://example.com/myproject/download/wss/jobs?hdr=1
      id grid alg status gridid args griddata tag creation_time metajobid
      093fd32b-4604-4c06-9ba9-d3aae7a6b80d Cloud cloud CANCEL  --image=ami-00000021   2012-09-11 11:20:06 
      0bba95f9-38ea-4cf9-8125-6bc3e93af154 Cloud cloud CANCEL  --image=ami-00000021   2012-09-13 10:40:07
      ...

      SOAP equivalent: —

      Querying a single job
      http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d
      093fd32b-4604-4c06-9ba9-d3aae7a6b80d Cloud cloud CANCEL  --image=ami-00000021   2012-09-11 11:20:06

      Any subset of the attributes can be selected. Consider the following SOAP commands:

      wsclient -m status -e http://example.com:8091/ -j 093fd32b-4604-4c06-9ba9-d3aae7a6b80d

      REST equivalent:

      http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d/id+status
      093fd32b-4604-4c06-9ba9-d3aae7a6b80d CANCEL
      wsclient -m griddata -e http://example.com:8091/ -j 093fd32b-4604-4c06-9ba9-d3aae7a6b80d

      REST equivalent:

      http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d/id+griddata
      093fd32b-4604-4c06-9ba9-d3aae7a6b80d d2048dd6-464e-47c9-a4d0-373a9de25bdb_e933a6f8-7365-4b58-846d-a38be7d4ba24_76a26b38-88fd-4ba3-912b-2b1e30ed74d6
      Querying finished jobs
      http://example.com/myproject/download/wss/jobs/finished/id+status
      093fd32b-4604-4c06-9ba9-d3aae7a6b80d FINISHED
      0bba95f9-38ea-4cf9-8125-6bc3e93af154 ERROR
      0f25e8cb-45f6-4f7f-83b4-4525dc245b11 FINISHED

      SOAP equivalent:

      wsclient -m finished -e http://example.com:8091/
      Cancelling a job or deleting a finished job

      To cancel a job, send a request to its URL with the DELETE verb.

      curl -X DELETE  http://example.com/myproject/download/wss/jobs/093fd32b-4604-4c06-9ba9-d3aae7a6b80d

      SOAP equivalent:

      wsclient -m delete -e http://example.com:8091/ -j 093fd32b-4604-4c06-9ba9-d3aae7a6b80d
      Submitting a job

      A job can be submitted to the Bridge by POSTing a job object to the jobs entity. Currently, there are two ways to do this; either by posting form-data (Content-Type: application/x-www-form-urlencoded) or by posting a JSON object (application/json). The following examples show how this can be achieved using curl.

      In both cases, by default, after a successful submission, the Bridge will respond with a redirect header to the newly created jobs (Location: http://example.com/myproject/download/wss/jobs/<new_uuid>). The -L switch tells curl to follow this redirection.

      Another possibility is to specify 'redir=0' in the URL (…/wss/jobs?redir=0); in this case, no redirection occurs, only the unique id of the new job is printed. This is the same behaviour as that of the SOAP interface.

      To submit a job, the same information has to be supplied as in case of the SOAP interface. Notice that—unlike the SOAP interface—the URL, MD5 and size parts of a specification are specified separately, not in a single string joined with '=' .

      Submitting using form data
      Submitting a job using CURL
      curl \
          -d name=dsp \
          -d grid=NULL \
          -d args='-dpf=docking.dpf' \
          -d tag='Test job' \
          -d input[docking.dpf][url]=http://example.com/myproject/download/inputs/publicautodock423/docking.dpf \
          -d input[docking.dpf][md5]=a7bac5a941a85660cd87bf83f93b852a \
          -d input[docking.dpf][size]=2933 \
          -d input[inputs.zip][url]=http://example.com/myproject/download/inputs/publicautodock423/inputs.zip \
          -d input[inputs.zip][md5]=cbb8c0ada8e197d6db707e6ce4f03ca6 \
          -d input[inputs.zip][size]=3270896 \
          -d output[0]=log.dlg \
          http://example.com/myproject/download/wss/jobs?hdr=1 -L
      id grid alg status gridid args griddata tag creation_time metajobid userid error_info
      022a9b75-8a08-4f5d-8188-c8687659e60d NULL dsp INIT  -dpf=docking.dpf  Test job 2013-01-22 13:30:32   

      Alternatively:

      Submitting a job using CURL
      curl \
          # [ same as above ] \
          http://example.com/myproject/download/wss/jobs?redir=0
      022a9b75-8a08-4f5d-8188-c8687659e60d
      Submitting using JSON

      This is an example job description in JSON:

      example-job.json
      [
          {
              "name": "dsp",
              "grid": "NULL",
              "args": "-dpf=docking.dpf",
              "tag": "Test job",
              "input": {
                  "docking.dpf": {
                      "url": "http://example.com/myproject/download/inputs/publicautodock423/docking.dpf",
                      "md5": "a7bac5a941a85660cd87bf83f93b852a",
                      "size": 2933
                  },
                  "inputs.zip": {
                      "url": "http://example.com/myproject/download/inputs/publicautodock423/inputs.zip",
                      "md5": "cbb8c0ada8e197d6db707e6ce4f03ca6",
                      "size": 3270896
                  }
              },
              "output": [
                  "log.dlg"
              ]
          }
      ]

      This example job can be submitted with the following command.

      Submitting a JSON job
      curl \
          --header 'Content-Type: application/json' \
          --data @example-job.json \
          http://example.com/myproject/download/wss/jobs
      [{"id":"98cba344-80cd-451a-bca8-cd3c887d0731"}]
      If plain text output is required
      curl \
          --header 'Content-Type: application/json' \
          --data @example-job.json \
          http://example.com/myproject/download/wss/jobs?format=plain
      af3b0de8-c2cc-43eb-ba27-67ae1cda27db

      Admin Guide

      Follow these steps to deploy the REST interface for the Bridge:

      1. Create a directory that will be published by the project web server. For example, if you create a directory named wss in project/download, the directory will be available through http://<projecturl>/download/wss.
      2. Setup the project site to allow RewriteEngine and security overrides in .htaccess files.
      3. Allow write access to the Bridge output directory.
      4. Setup security.
      5. Copy the php scripts that constitute the interface to the directory just created.
      6. Configure the interface by setting the necessary values in 3gb.php.
      Create the directory
      mkdir $HOME/project/download/wss
      Setup Apache

      Setup Apache and the project site to support the Rewrite module and security overrides.

      Enabling the Rewrite engine as ''root''
      sudo a2enmod rewrite

      In the project site configuration ([project_home]/project/[project_name].httpd.conf), FileInfo and Limit has to be added to AllowOverride for directory …/user. This enables the use of RewriteEngine and security overrides in .htaccess.

      As output directories will be created upon submission, PHP must be allowed to access the 3G Bridge output directory (as configured in 3gbridge.conf/[wssubmitter]/output-dir). To do so, add this directory, delimited with a colon, to the open_basedir PHP admin value.

      [project_home]/project/[project_name].httpd.conf
      [...]
      <Directory "[project_home]/project/html/user">
         [...]
          AllowOverride AuthConfig FileInfo Limit
         [...]
          php_admin_value "open_basedir"    "...:[project_home]/3g-bridge/output"
      [...]   

      Restart Apache for these changes to take effect:

      sudo service apache2 restart
      Allow write access to the Bridge output directory

      Assuming the Bridge output directory is [project_home]/master/3g-bridge/output. The actual value can be found in the 3G Bridge configuration file, in the [wssubmitter] section.

      Make sure that the Apache process user (usually www-data) is added to the project user's default group

      getent group $USER
      boinc-home:x:108:www-data

      If www-data is not listed next to the project user default group, you must correct this:

      sudo usermod -a -G $USER www-data

      When done, add group write access to the output directory and its direct descendants:

      Make the output directories writable to the Apache user
      cd $HOME/master/3g-bridge
      chmod 775 `find output -maxdepth 1 -type d`

      Create a logfile, which is writable by the Apache user:

      Create a log file writable by the Apache user
      LOGFILE="$HOME/project/log_<dir>/restinterface.log"
      touch "$LOGFILE"
      chmod g+w "$LOGFILE"
      Configure Apache for the REST interface

      The following is an example .htaccess file we use, which enables the use of “pretty” URLs, and restricts access to our internal network. You can setup any kind of security Apache supports.

      [project_home]/project/download/wss/.htaccess
      Order deny,allow
      Deny from all
      Allow from lpds.sztaki.hu
       
      <IfModule mod_rewrite.c>
      RewriteEngine On
      RewriteBase /[project_name]/download/wss/
      RewriteCond %{REQUEST_FILENAME} !-f
      RewriteCond %{REQUEST_FILENAME} !-d
      RewriteRule ^(.*)$ 3gb.php/$1 [L]
      </IfModule>
      Configure the interface

      To configure the interface, set the following values in config.inc.php. Usually, this involves only the following steps: (1) create a hard link to the Bridge configuration in the wss directory, (2) set the base URL of the REST interface, and (3) set the path of the log file. For completeness, other possibilities are also detailed here.

      Set the base URL of the web service. It is required for redirects.

      Set the path of the log file. This file must be writable by the Apache user.

      Set the log level. Only messages with level at least this high will be logged. Possible settings—in order, highest to lowest—are:

      1. CRITICAL – unrecoverable errors, unhandled exceptions.
      2. ERROR – handled, but notable errors.
      3. AUDIT – produces exactly two entries for each request:
        1. For security reasons, the fact that a request is being handled, including information on the client.
        2. A single line summarising what happened.
      4. INFO – Detailed information on what happened.
      5. DEBUG – Debug information useful for developers.
      6. TRACE – Debug information useful for nobody.

      For security reasons, we advise you to never set the level higher than AUDIT. Lower settings may produce extremely large amount of log data.

      The CONFIG_FILE constant must point to the 3G Bridge config file. If Apache does not allow to follow symlinks or to access files anywhere on the filesystem, create a hard link to it in the wss dir.

      Create a hard link to the 3G Bridge config file
      cd [project_home]/project/download/wss
      ln [project_home]/master/3g-bridge/3g-bridge.conf

      The Bridge version is queried by executing BRIDGE_VER_CMD. Make sure the path it uses points to the 3g-bridge binary. The default setting is usually right.

      Set the destination grid for meta-jobs (the name of the grid defined in 3g-bridge.conf). The default value is usually right.

      Example: [project_home]/project/download/wss/config.inc.php
      define('BASE_URL', 'https://PROJECT_URL/download/wss');
      define('LOG_FILE', '/var/lib/boinc/home/project/log_home/restif.log');
      define('LOG_LEVEL', 'AUDIT');
      define('CONFIG_FILE', '3g-bridge.conf');
      define('BRIDGE_VER_CMD',
             '/bin/bash -c /usr/sbin/3g-bridge\ -V');
      define('METAJOB_GRID', 'Metajob');
      manual/restif.txt · Last modified: 2013/05/13 12:16 (external edit)