- Simple scenarios
- Complex scenarios
- Bridging scenarios
- Basic components
- Application support
- Job submission interface alternatives
- Additional components
- Basic components
- Application support
- Additional components
Table of Contents
3G Bridge Installation
This manual describes the installation and configuration of the 3G Bridge with BOINC integration (DC-API).
The 3G Bridge and BOINC have been developed for Debian 6 systems. Although the Brigde is known to work with Ubuntu and Scientific Linux 6, these systems are different from Debian. Ubuntu is only slightly different, and most of the scripts listed here will work as-is, or with little modification. In SL6, however, the scripts and the list of packages needed to be installed are very different, and are not covered here.
To use the 3G Bridge as a front-end for BOINC, a working BOINC project has to be created before the Bridge can be configured.
The Bridge packages can be found in the SZDG repositories. Also, the Bridge uses DC-API as an interface to BOINC, which is also found in this repository. Whether you are installing the Bridge from packages or compiling the Bridge from source, you will have to have the SZDG repository in your APT sources list.
# As root echo 'deb http://www.desktopgrid.hu/debian/ squeeze szdg' >> /etc/apt/sources.list apt-key advanced --keyserver keys.niif.hu --recv-keys 6A2165907B1AAC6F apt-get update
The 3G Bridge can be easily installed from packages. The installer will ask whether you want to use the Bridge in standalone mode. As you will use the Bridge with BOINC, Select NO.
wsclientis not mandatory on the server, but it is useful for testing job submission.
# As root apt-get install 3g-bridge 3g-bridge-wsclient
The Bridge can be installed from source if absolutely necessary, but installing the Bridge from the package repository is the preferred way.
Integrating the 3G Bridge with BOINC
At this point, there is a working BOINC project on the server, and the 3G Bridge has been installed and is ready to be used as a BOINC master application. We assume that 1) the Bridge has been installed from packages, and 2) the project is called
test. When using this manual, don't forget to change the BOINC username, the database name, etc. to match your actual project name.
Preparing the Database
Switch to the BOINC project user.
To use the Bridge as a master application, you must first create the 3G Bridge database. You can simply create the 3G Bridge tables in the already existing BOINC database 1). Provide username and password to mysql as necessary.
export PNAME=test # project name sudo su - boinc-$PNAME stop # don't forget to stop the project first mysql boinc_$PNAME < /usr/share/3g-bridge/schema.sql
Creating Job Queues
If you have no client applications installed in BOINC yet, you can skip this step for now.
You can add or remove queues any time in the future, provided you stop the project first.
Each queue has two defining attributes:
gridA single Bridge can be the entry point for multiple back-end infrastructures—the
gridattribute refers to one of these back-ends. The value of this attribute must match one of the queue definitions in the 3G Bridge configuration.
algThe name of the application in the
grid. The value of this attribute must match an installed BOINC application's name.
algattributes together define a single queue. You have to define a queue for each application registered in BOINC.
The 3rd attribute of a queue (
batchsize) defines the number of jobs the Bridge will process in a single iteration. The larger this value is, the better the throughput of the 3G Bridge will be for a single queue. On the other hand, a larger
batchsizewill consume more memory, may increase the load of the system, and, if you have many queues, it will increase the wait time of other queues. Generally, 10 is an acceptable default value for this attribute.
mysql boinc_$PNAME -e \ "insert into cg_algqueue (grid, alg, batchsize) values ('gridname', 'algname', 10)"
Adding the 3G Bridge to BOINC
After creating the 3G Bridge database, you can add the Bridge to BOINC as a master application. A new directory will be created in the project home directory:
'$HOME/master/3g-bridge/'. This directory contains all Bridge configuration files, which will be covered in the next section.
boinc_appmgr --add --master /usr/share/3g-bridge/master-ws.xml
The configuration information of the Bridge is in the file
$HOMEis the project home directory. This file is a glib ini file, which defines key-value pairs grouped in sections. The configuration file and the values in it are case sensitive. The semantics of the configuration file is described in comments in the file and detailed in the 3G Bridge manual (
man 5 3g-bridge.conf). Here, we will only cover the necessary steps to initally configure the 3G Bridge.
The following listing shows an excerpt from a
3g-bridge.conffile. The keys included in this listing are those that must be set.
[database] host = localhost name = boinc_<projectname> user = boinc_<projectname> password = <found in $HOME/project/config.xml> [bridge] log-target = stdout pid-file = /var/lib/boinc/<projectname>/master/3g-bridge/bridge.pid [wsmonitor] log-target = stdout pid-file = /var/lib/boinc/<projectname>/master/3g-bridge/monitor.pid port = <8092> [wssubmitter] port = <8091> log-target = stdout pid-file = /var/lib/boinc/<projectname>/master/3g-bridge/wssubmitter.pid input-dir = /var/lib/boinc/<projectname>/master/3g-bridge/input output-dir = /var/lib/boinc/<projectname>/project/download/bridge-output output-url-prefix = http://<project_url>/download/bridge-output [<gridname>] handler = DC-API-Single dc-api-config = /var/lib/boinc/<projectname>/master/3g-bridge/dc-api.conf [dlmgr] handler=DownloadManager download-threads=10 ...
The following list details the modifications you have to make in the default configuration file. The list is not ordered, and it's very detailed; we recommend you to only skim through it before modifying the configuration file to know what you will be doing, and then use the listing above as a guide for modifying your configuration.
[defaults]section contains default and general setting, used by all or any component of the Bridge.
log-levelas you need to. It may be pratical to set it to
DEBUGfor a while after the installation.
monlog-*keys are detailed in the 3G Bridge monitoring page.
[database]section so the Brigde will be able to access the database. Typical values are shown in the listing, change them as necessary. The password can be found in
These settings apply to all components (active processes) of the Bridge:
log-targetsetting for all components should be
stdout. In this case, the Bridge logs will appear in the standard log directory of the project:
pid-filesetting for all components to a path that they can use. Set different file names for each component, and make sure the specified files can be createted and/or overwritten. See the listing for an example.
output-dirsettings must point to existing directories. So don't forget to create these directories, if they don't exist. The Bridge must have write permission on both directories.
INDIR=/var/lib/boinc/<projectname>/master/3g-bridge/input OUTDIR=/var/lib/boinc/<projectname>/project/download/bridge-output mkdir -p "$INDIR" "$OUTDIR"
The output directory must be accessible through http. (It's usually a good practice to place the output directory in
$HOME/project/download.) Set the value of
output-url-prefixto the http URL through which the output directory is accessible. This is usually
If you have multiple projects, don't forget to set a separate port for each one. This can be any free port in the non-privileged range (>1024). If you have a single project and no other services running on the host, you can leave all ports with their default values.
Configuring plugins (queues)
Configure the queues of the 3G Bridge. Each additional section will configure a queue for the Bridge. The name of the section will be the name of the queue, which has to match the queue name specified in
disablesetting of a queue to
truecompletely disables it. Example queues in the default configuration are disabled by default, you can just ignore them.
handlersetting for a queue will define the plugin which handles that queue. Other configuration options in the plugin's section depend on which handler is configured.
Configuring the BOINC plugin
In case of BOINC, the plugin is
'DC-API-Single'(sans quotes). This plugin needs additional settings from the
dc-api.conffile. The path to
dc-api.confmust be specified with the setting
dc-api-config. This file is usually in the same directory as the
3g-bridge.conf. If you are using SZDG, you may not need to change the
The settings shown on the listing above are typical; changing only the parts denoted by
< >s is usually sufficient. Don't forget to:
- Create the input and output directories.
- Make sure that pid-files can be created and their directories exist.
- Set different
wsmonitorports for different projects (iff you have multiple projects).
Verifying the Installation
After configuring the 3G Bridge for BOINC, you can start the project with
start. After this command has finished, check the status of the project, and make sure the 3G Bridge components are listed as running.
If all components are running, check the log files in
'$HOME/project/log_<hostname>/'whether everything works normally. The filenames of the log files will match the
commandlinefield in the output of the
start && status
... DAEMON pid status lockfile disabled commandline 1 4405 running locked no feeder -d 3 2 4407 running locked no transitioner -d 3 3 4410 running locked no file_deleter -d 3 4 4413 running locked no app_3g-bridge_3g-bridge_1 -f -c /var/li... 5 4415 running locked no app_3g-bridge_wssubmitter_2 -f -c /var/... 6 4433 running locked no app_3g-bridge_wsmonitor_3 -f -c /var/li...
Finally, make sure that the Bridge is actually reachable. To do so, try querying its version from the host you will usually submit from.
HOST=<project hostname> PORT=<port specified in the [wssubmitter] section of the config file> wsclient -e http://$HOST:$PORT/ -m version
When adding repository key, apt-key hangs.
Open a browser and go to http://pools.keyserver.eu/, where you will find other possible keyservers. Choose and try any one of them in place of
When executing ''wsclient -m version'', I receive the following error: 'Validation constraint violation: tag name or namespace mismatch'.
This happens when the
wssubmitterwas compiled with a gSOAP 2.8.3 or 2.8.4 package installed. These versions of gSOAP are known to produce buggy C++ code in this particular case (
If the current version of gSOAP in your package repository is one of these, you can download and compile another version of gSOAP. We have verified that version 2.7.9 and 2.8.8 work as expected.
If you have all the dependencies installed, it may be easier to download the 2.7.9 package from the Debian squeeze repositories. Make sure upgrades are held back, until 2.8.8 rolls out for your system:
# As root EXAMPLEURL=http://ftp.us.debian.org/debian/pool/main/g/gsoap/gsoap_2.7.9l-0.2_amd64.deb wget "$EXAMPLEURL" dpkg -i gsoap_2.7.9l-0.2_amd64.deb echo 'gsoap hold' | dpkg --set-selections
When executing ''wsclient'', I get connection REFUSED error.
wssubmitteris not running. Try restarting the project, and check the logs for information on the problem.
When executing ''wsclient'', I get connection TIMED OUT error.
This usually means that there's a firewall denying the connection. Another explanation can be that the port is open, but the Bridge cannot answer incoming connections. This can be caused by 1) high load on the project server, 2) too many open connections or high network traffic, 3) the Bridge has failed silently. In the latter case, you should just restart the project, and check the logs for information on the problem.
When submitting a job with ''wsclient'', I get the error End of file or no input.
Upon submission, the
wsclientproduces the following error:
Error -1 fault: SOAP-ENV:Client [no subcode] "End of file or no input: Operation interrupted or timed out" Detail: [no detail]
This usually means that either the input or the output directory has not been created on the server. See the [wssubmitter] configuration section.manual/3gbridge.txt · Last modified: 2013/02/27 13:29 (external edit)