Monitoring Facility for the 3G Bridge

There is a built-in monitoring facility in the 3G Bridge, called MonLog. If enabled, the Bridge will log specific events to the file specified in the configuration. These events are:

  • Job arrival to the Bridge (→INIT)
  • Job submitted to back-end grid (INIT→RUNNING)
  • Job finished successfully (RUNNING→FINISHED)

Periodically, this file is renamed (rotated) and a new file is started. There is no garbage collection, old files have to be deleted from time to time (manually or using cron for example).

The configuration options of this feature are in the [defaults] section of the configuration file.

3g-bridge.conf:
 
[defaults]
 
...
 
monlog-target = /var/log/3g-bridge/monitor.log
monlog-format = xml
monlog-timezone = CET
monlog-rotate-interval = 10
monlog-rotate-filenamefmt = report_{ts}.xml
monlog-semaphore-name = default
...
  • monlog-target: The path of the buffer file for log output. This feature can be turned off by setting monlog-target to an empty string. If set, the base directory must exist and must be writable.
  • monlog-format: Either xml or simple. Controls the output format of the messages. Examples can be found below.
  • monlog-timezone: String, included in the output messages if the output format is xml. Its not interpreted.
  • monlog-rotate-interval: Time to elapse, in minutes, between log rotations.
  • monlog-rotate-filenamefmt: After log file rotation, the resulting files will be named according to this option. The literal {ts} will be replaced with the current UNIX timestamp.
  • monlog-semaphore-name: A semaphore is used by separate processes of the Bridge using the buffer file to avoid simultaneous writing. This string is included in the semaphore's name. If multiple projects are running on the same host, this string must be used to separate Bridges of different projects.1)

When an event is logged, it is written in the file specified by monlog-target. Consecutive log messages are gathered in this buffer file until several minutes—specified by monlog-rotate-interval—has elapsed; when, the buffer file is renamed according to monlog-rotate-filenamefmt, and a new buffer is started. According to the configuration above, the messages are gathered in /var/lib/3g-bridge/monitor.log; and every hour, the buffer is renamed to /var/lib/3g-birdge/result-XXX.log where XXX is replaced with the UNIX timestamp upon renaming.

Based on monlog-format, the monitoring feature can produce two kinds of logs, which are shown on the following examples.

<?xml version="1.0" encoding="UTF-8"?>
<report timestamp="1358427736237" timezone="CET" version="1.1">
  <metric_data>
    <dt>2013-01-17 14:00:17</dt>
    <event>job_entry</event>
    <job_id>eca57217-2c5b-4647-bdc9-852b7d4cb72c</job_id>
    <application>dsp</application>
  </metric_data>
  <metric_data>
    <dt>2013-01-17 14:00:20</dt>
    <event>job_submission</event>
    <job_id>eca57217-2c5b-4647-bdc9-852b7d4cb72c</job_id>
    <output_grid_name>edgidemo</output_grid_name>
  </metric_data>
  <metric_data>
    <dt>2013-01-17 14:00:20</dt>
    <event>job_status</event>
    <job_id>eca57217-2c5b-4647-bdc9-852b7d4cb72c</job_id>
    <status>Running</status>
  </metric_data>
  <metric_data>
    <dt>2013-01-17 14:01:10</dt>
    <event>job_status</event>
    <job_id>b1390a4e-b600-4401-95ff-d89ce5005eef</job_id>
    <status>Finished</status>
  </metric_data>
</report>
dt=2011-04-03_00:19:43 event=job_entry job_id=fe91e44e-3ea4-43cb-88aa-1958ccb769dc application=dsp
dt=2011-04-03_00:26:04 event=job_submission job_id=fe91e44e-3ea4-43cb-88aa-1958ccb769dc job_id_dg=f33670db-4f01-4875-9ebb-b280c17bd9d3_821fe28e-d42d-4fea-b206-a1e2d887c44b output_grid_name=edgidemo
dt=2011-04-03_00:26:04 event=job_status job_id=fe91e44e-3ea4-43cb-88aa-1958ccb769dc status=Running
dt=2011-04-04_22:55:06 event=job_status job_id=47aacb82-e4fd-417f-a29b-b793f064d57f status=Finished
1) In the current development version, all log writing has been moved to the core bridge process, making the use of semaphores unnecessary; therefore, making this option obsolete.
manual/3gbplugin-monitor.txt · Last modified: 2013/07/08 13:54 by smith
Trace: 3gbplugin-monitor
Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0