User Tools

Site Tools


manual:restif

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
manual:restif [2013/01/22 10:53]
a.visegradi
manual:restif [2013/05/13 12:16] (current)
Line 14: Line 14:
 In this manual, we assume that the project URL is In this manual, we assume that the project URL is
 ''​http:​%%//​%%example.com/​myproject/'',​ and the REST interface is ''​http:​%%//​%%example.com/​myproject/'',​ and the REST interface is
-[[#AdminGuide|deployed]] in ''​download/​wss/''​. That is, the base URL of the REST+[[#Admin_Guide|deployed]] in ''​download/​wss/''​. That is, the base URL of the REST
 interface is ''​%%http://​example.com/​myproject/​download/​wss/​%%''​. interface is ''​%%http://​example.com/​myproject/​download/​wss/​%%''​.
  
Line 22: Line 22:
 arrays). This response will be rendered according to the ''​format''​ argument, arrays). This response will be rendered according to the ''​format''​ argument,
 which can be ''​plain'',​ ''​html''​ or ''​json''​. The ''​hdr''​ argument enables (1) which can be ''​plain'',​ ''​html''​ or ''​json''​. The ''​hdr''​ argument enables (1)
-or disables (0) the rendering of a header if applicable. ​The default format is +or disables (0) the rendering of a header if applicable. ​If the Content-Type of the request is ''​ '​application/​json'​ '',​ the response will be in ''​JSON''​ format. Otherwise, the default ​response ​format is ''​plain''​.
-''​plain''​.+
  
   * **''​plain''​**:​ The response will be rendered as plain text. Records are separated with '​\n',​ attributes are separated with a single space. Separators can be overridden with the ''​sep''​ and ''​rsep''​ (record sepatator) arguments. A header is rendered if requested (''​hdr=1''​). ​   * **''​plain''​**:​ The response will be rendered as plain text. Records are separated with '​\n',​ attributes are separated with a single space. Separators can be overridden with the ''​sep''​ and ''​rsep''​ (record sepatator) arguments. A header is rendered if requested (''​hdr=1''​). ​
Line 30: Line 29:
  
 Example: Example:
-  http://​example.com/​download/​wss/​version+  http://​example.com/myproject/​download/​wss/​version
 <​file>​ <​file>​
 3G Bridge 1.9 3G Bridge 1.9
 </​file>​ </​file>​
-  http://​example.com/​download/​wss/​version?​hdr=1+  http://​example.com/myproject/​download/​wss/​version?​hdr=1
 <​file>​ <​file>​
 version version
 3G Bridge 1.9 3G Bridge 1.9
 </​file>​ </​file>​
-  http://​example.com/​download/​wss/​version?​format=json+  http://​example.com/myproject/​download/​wss/​version?​format=json
 <​file>​ <​file>​
 [{"​version":"​3G Bridge 1.9"}] [{"​version":"​3G Bridge 1.9"}]
Line 47: Line 46:
 argument. The record separator can be overridden with ''​rsep''​. argument. The record separator can be overridden with ''​rsep''​.
  
-  http://​example.com/​download/​wss/​jobs/​093fd32b-4604-4c06-9ba9-d3aae7a6b80d?​sep=|+  http://​example.com/myproject/​download/​wss/​jobs/​093fd32b-4604-4c06-9ba9-d3aae7a6b80d?​sep=|
 <​file>​ <​file>​
 093fd32b-4604-4c06-9ba9-d3aae7a6b80d|Cloud|cloud|CANCEL||--image=ami-00000021|||2012-09-11 11:20:06| 093fd32b-4604-4c06-9ba9-d3aae7a6b80d|Cloud|cloud|CANCEL||--image=ami-00000021|||2012-09-11 11:20:06|
Line 55: Line 54:
 the URL: the URL:
  
-  http://​example.com/​download/​wss/​jobs/​093fd32b-4604-4c06-9ba9-d3aae7a6b80d/​id+status+creation_time+  http://​example.com/myproject/​download/​wss/​jobs/​093fd32b-4604-4c06-9ba9-d3aae7a6b80d/​id+status+creation_time
 <​file>​ <​file>​
 93fd32b-4604-4c06-9ba9-d3aae7a6b80d CANCEL 2012-09-11 11:20:06 93fd32b-4604-4c06-9ba9-d3aae7a6b80d CANCEL 2012-09-11 11:20:06
Line 180: Line 179:
 == Submitting a job == == Submitting a job ==
  
-FIXME+A job can be submitted to the Bridge by ''​POST''​ing a job object to the ''​jobs''​ 
 +entity. Currently, there are two ways to do this; either by posting form-data 
 +(''​Content-Type:​ application/​x-www-form-urlencoded''​) or by posting a JSON 
 +object (''​application/​json''​). The following examples show how this can be 
 +achieved using **''​curl''​**.
  
 +In both cases, by default, after a successful submission, the Bridge will
 +respond with a redirect header to the newly created jobs
 +(''​%%Location:​ http://​example.com/​myproject/​download/​wss/​jobs/<​new_uuid>​%%''​). The
 +''​-L''​ switch tells ''​curl''​ to follow this redirection.
 +
 +Another possibility is to specify ''​ '​redir=0'​ ''​ in the URL
 +(''​.../​wss/​jobs?​redir=0''​);​ in this case, no redirection occurs, only the unique
 +id of the new job is printed. This is the same behaviour as that of the SOAP
 +interface.
 +
 +To submit a job, the same information has to be supplied as in case of the
 +[[manual:​soapif|SOAP interface]]. Notice that---unlike the SOAP interface---the
 +URL, MD5 and size parts of a specification are specified separately, not in a
 +single string joined with ''​ '​='​ ''​.
 +
 +== Submitting using form data ==
 +
 +<code bash Submitting a job using CURL>
 +curl \
 +    -d name=dsp \
 +    -d grid=NULL \
 +    -d args='​-dpf=docking.dpf'​ \
 +    -d tag='​Test job' \
 +    -d input[docking.dpf][url]=http://​example.com/​myproject/​download/​inputs/​publicautodock423/​docking.dpf \
 +    -d input[docking.dpf][md5]=a7bac5a941a85660cd87bf83f93b852a \
 +    -d input[docking.dpf][size]=2933 \
 +    -d input[inputs.zip][url]=http://​example.com/​myproject/​download/​inputs/​publicautodock423/​inputs.zip \
 +    -d input[inputs.zip][md5]=cbb8c0ada8e197d6db707e6ce4f03ca6 \
 +    -d input[inputs.zip][size]=3270896 \
 +    -d output[0]=log.dlg \
 +    http://​example.com/​myproject/​download/​wss/​jobs?​hdr=1 -L
 +</​code>​
 +<​file>​
 +id grid alg status gridid args griddata tag creation_time metajobid userid error_info
 +022a9b75-8a08-4f5d-8188-c8687659e60d NULL dsp INIT  -dpf=docking.dpf ​ Test job 2013-01-22 13:​30:​32 ​  
 +</​file>​
 +
 +Alternatively:​
 +
 +<code bash Submitting a job using CURL>
 +curl \
 +    # [ same as above ] \
 +    http://​example.com/​myproject/​download/​wss/​jobs?​redir=0
 +</​code>​
 +<​file>​
 +022a9b75-8a08-4f5d-8188-c8687659e60d
 +</​file>​
 +
 +== Submitting using JSON ==
 +
 +This is an example job description in JSON:
 +
 +<file json example-job.json>​
 +[
 +    {
 +        "​name":​ "​dsp",​
 +        "​grid":​ "​NULL",​
 +        "​args":​ "​-dpf=docking.dpf",​
 +        "​tag":​ "Test job",
 +        "​input":​ {
 +            "​docking.dpf":​ {
 +                "​url":​ "​http://​example.com/​myproject/​download/​inputs/​publicautodock423/​docking.dpf",​
 +                "​md5":​ "​a7bac5a941a85660cd87bf83f93b852a",​
 +                "​size":​ 2933
 +            },
 +            "​inputs.zip":​ {
 +                "​url":​ "​http://​example.com/​myproject/​download/​inputs/​publicautodock423/​inputs.zip",​
 +                "​md5":​ "​cbb8c0ada8e197d6db707e6ce4f03ca6",​
 +                "​size":​ 3270896
 +            }
 +        },
 +        "​output":​ [
 +            "​log.dlg"​
 +        ]
 +    }
 +]
 +</​file>​
 +
 +This example job can be submitted with the following command.
 +
 +<code bash Submitting a JSON job>
 +curl \
 +    --header '​Content-Type:​ application/​json'​ \
 +    --data @example-job.json \
 +    http://​example.com/​myproject/​download/​wss/​jobs
 +</​code>​
 +<file json>
 +[{"​id":"​98cba344-80cd-451a-bca8-cd3c887d0731"​}]
 +</​file>​
 +
 +<code bash If plain text output is required>​
 +curl \
 +    --header '​Content-Type:​ application/​json'​ \
 +    --data @example-job.json \
 +    http://​example.com/​myproject/​download/​wss/​jobs?​format=plain
 +</​code>​
 +<​file>​
 +af3b0de8-c2cc-43eb-ba27-67ae1cda27db
 +</​file>​
 ===== Admin Guide ===== ===== Admin Guide =====
  
Line 187: Line 289:
  
   - Create a directory that will be published by the project web server. For example, if you create a directory named ''​wss''​ in ''​project/​download'',​ the directory will be available through ''​%%http://<​projecturl>/​download/​wss%%''​.   - Create a directory that will be published by the project web server. For example, if you create a directory named ''​wss''​ in ''​project/​download'',​ the directory will be available through ''​%%http://<​projecturl>/​download/​wss%%''​.
-  - Setup **security** for that directory. Create a ''​.htaccess'' ​file in the directory, and enable overriding ''​AuthConfig''​ in the project'​s ''​httpd.conf''​+  - Setup the project site to allow RewriteEngine and security ​overrides in ''​.htaccess'' ​files. 
-  - Simply copy the ''​php''​ scripts that constitute the interface to the directory just created.+  - Allow **write access** to the Bridge output ​directory. 
 +  - Setup **security**
 +  - Copy the ''​php''​ scripts that constitute the interface to the directory just created
 +  - Configure the interface by setting the necessary values in ''​3gb.php''​.
  
-The following is an example ''​.htaccess''​ file that enables the use of "​pretty"​ +== Create the directory == 
-URLs, and restricts access to internal network. You can setup any kind of+ 
 +<code bash> 
 +mkdir $HOME/​project/​download/​wss 
 +</​code>​ 
 + 
 +== Setup Apache == 
 + 
 +Setup Apache and the project site to support the Rewrite module and security overrides. 
 + 
 +<code bash Enabling the Rewrite engine as ''​root''>​ 
 +sudo a2enmod rewrite 
 +</​code>​ 
 + 
 +In the project site configuration (''​[project_home]/​project/​[project_name].httpd.conf''​),​ ''​FileInfo''​ and ''​Limit''​ has to be added to ''​AllowOverride''​ for directory ''​.../​user''​. This enables the use of RewriteEngine and security overrides in ''​.htaccess''​. 
 + 
 +As output directories will be created upon submission, PHP must be allowed to access the 3G Bridge output directory (as configured in ''​3gbridge.conf/​[wssubmitter]/​output-dir''​). To do so, add this directory, delimited with a colon, to the ''​open_basedir''​ PHP admin value. 
 + 
 +<file apache [project_home]/​project/​[project_name].httpd.conf>​ 
 +[...] 
 +<​Directory "​[project_home]/​project/​html/​user">​ 
 +   ​[...] 
 +    AllowOverride AuthConfig FileInfo Limit 
 +   ​[...] 
 +    php_admin_value "​open_basedir" ​   "​...:​[project_home]/​3g-bridge/​output"​ 
 +[...]    
 +</​file>​ 
 + 
 +Restart Apache for these changes to take effect: 
 + 
 +<code bash> 
 +sudo service apache2 restart 
 +</​code>​ 
 + 
 +== Allow write access to the Bridge output directory == 
 + 
 +Assuming the Bridge output directory is ''​[project_home]/​master/​3g-bridge/​output''​. The actual value can be found in the 3G Bridge configuration file, in the ''​[wssubmitter]''​ section. 
 + 
 +Make sure that the Apache process user (usually ''​www-data''​) is added to the project user's default group 
 + 
 +<code bash> 
 +getent group $USER 
 +</​code>​ 
 +<​file>​ 
 +boinc-home:​x:​108:​www-data 
 +</​file>​ 
 + 
 +If ''​www-data''​ is not listed next to the project user default group, you must correct this: 
 + 
 +<code bash> 
 +sudo usermod -a -G $USER www-data 
 +</​code>​ 
 + 
 +When done, add group write access to the output directory and its direct descendants:​ 
 + 
 +<code bash Make the output directories writable to the Apache user> 
 +cd $HOME/​master/​3g-bridge 
 +chmod 775 `find output -maxdepth 1 -type d` 
 +</​code>​ 
 + 
 +Create a logfile, which is writable by the Apache user: 
 + 
 +<code bash Create a log file writable by the Apache user> 
 +LOGFILE="​$HOME/​project/​log_<​dir>/​restinterface.log"​ 
 +touch "​$LOGFILE"​ 
 +chmod g+w "​$LOGFILE"​ 
 +</​code>​ 
 + 
 +== Configure Apache for the REST interface == 
 + 
 +The following is an example ''​.htaccess''​ file we use, which enables the use of "​pretty"​ 
 +URLs, and restricts access to our internal network. You can setup any kind of
 security Apache supports. security Apache supports.
  
-<file apache .htaccess>​+<file apache ​[project_home]/​project/​download/​wss/​.htaccess>​
 Order deny,allow Order deny,allow
 Deny from all Deny from all
Line 201: Line 376:
 <​IfModule mod_rewrite.c>​ <​IfModule mod_rewrite.c>​
 RewriteEngine On RewriteEngine On
-RewriteBase /edgidemo/​download/​wss/​+RewriteBase /[project_name]/​download/​wss/​
 RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-f
 RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-d
Line 208: Line 383:
 </​file>​ </​file>​
  
-Don't forget to enable overriding authorization in the project'​s main +== Configure ​the interface ==
-''​httpd.conf'':​+
  
-<file apache [project_home]/​project/​[project_name].httpd.conf> +To configure the interface, set the following values in ''​config.inc.php''​Usually, this involves only the following steps: (1) create a hard link to the Bridge configuration in the ''​wss''​ directory, (2) set the base URL of the REST interface, and (3) set the path of the log fileFor completeness,​ other possibilities are also detailed here.
-[...] +
-<​Directory "​[project_home]/​project/​html/​user">​ +
-   [...+
-    AllowOverride AuthConfig FileInfo Limit+
  
-[...]   ​+Set the base URL of the web service. It is required for redirects. 
 + 
 +Set the path of the log file. This file must be writable by the Apache user. 
 + 
 +Set the log level. Only messages with level at least this high will be logged. Possible settings---in order, highest to lowest---are:​ 
 +  - CRITICAL -- unrecoverable errors, unhandled exceptions. 
 +  - ERROR -- handled, but notable errors. 
 +  - AUDIT -- produces exactly two entries for each request: 
 +    - For security reasons, the fact that a request is being handled, including information on the client. 
 +    - A single line summarising what happened. 
 +  - INFO -- Detailed information on what happened. 
 +  - DEBUG -- Debug information useful for developers. 
 +  - TRACE -- Debug information useful for nobody. 
 + 
 +For security reasons, we advise you to never set the level higher than AUDIT. Lower settings may produce extremely large amount of log data. 
 + 
 +The ''​CONFIG_FILE''​ constant must point to the 3G Bridge config file. If Apache does not allow to follow symlinks or to access files anywhere on the filesystem, create a hard link to it in the ''​wss''​ dir. 
 + 
 +<code bash Create a hard link to the 3G Bridge config file> 
 +cd [project_home]/​project/​download/​wss 
 +ln [project_home]/​master/​3g-bridge/​3g-bridge.conf 
 +</​code>​ 
 + 
 +The Bridge version is queried by executing ''​BRIDGE_VER_CMD''​Make sure the path it uses points to the ''​3g-bridge''​ binaryThe default setting is usually right. 
 + 
 +Set the destination grid for meta-jobs (the name of the grid defined in ''​3g-bridge.conf''​). The default value is usually right. 
 + 
 +<file php Example: [project_home]/​project/​download/​wss/​config.inc.php>​ 
 +define('​BASE_URL',​ '​https://​PROJECT_URL/​download/​wss'​);​ 
 +define('​LOG_FILE',​ '/​var/​lib/​boinc/​home/​project/​log_home/​restif.log'​);​ 
 +define('​LOG_LEVEL',​ '​AUDIT'​);​ 
 +define('​CONFIG_FILE',​ '​3g-bridge.conf'​);​ 
 +define('​BRIDGE_VER_CMD',​ 
 +       '/​bin/​bash -c /​usr/​sbin/​3g-bridge\ -V'​);​ 
 +define('​METAJOB_GRID',​ '​Metajob'​);​
 </​file>​ </​file>​
manual/restif.1358852013.txt.gz · Last modified: 2013/01/22 10:53 by a.visegradi