summaryrefslogtreecommitdiffstats
path: root/doc/reports/dynamic.txt
blob: 38d4c7e3aa483348765050e8a262608db586dde8 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
.. -*- mode: rst -*-

.. _reports-dynamic:

==========================
Bcfg2 Web Reporting System
==========================

Summary and Features
====================

The new reporting system was implemented to address a number of
deficiencies in the previous system. By storing statistics data in a
relational database, we are now able to view and analyze more information
about the state of the configuration, including information about previous
configuration. Specific features in the new system include:

* The ability to look at a :ref:`reports-calendar-summary` with past
  statistics information.
* More recent data concerning hosts.
* Additional information display in reports. Primarily, reasons for
  :ref:`configuration item verification failure <reports-item-detail>`
  are now accessible.
* Instead of static pages, pages are generated on the fly, allowing
  users to drill down to find out about a :ref:`specific host
  <reports-client-detail>`, rather than only having one huge page with
  too much information.
* Ability to store reporting data separately from other server data.

Installation
============

Quickstart
----------

:ref:`appendix-guides-web-reports-install`

Prerequisites
-------------

* sqlite3
* pysqlite2 (if using python 2.4)
* `Django <http://www.djangoproject.com>`_ >= 1.3
* mod-wsgi

.. warning::

    There is a known issue when using an sqlite database on an ext4
    filesystem. You will want to remount the filesystem without barriers
    (-o barrier=0) in order to speed up the operations of the database.
    For more information, please see
    http://phoronix-test-suite.com/pipermail/trondheim-pts_phoronix-test-suite.com/2009-March/000095.html.

Install
-------


1. Be sure to include the specified fields included in the example
   ``bcfg2.conf`` file. These can be specified in either
   ``/etc/bcfg2.conf``, if it is readable by the webserver user,
   or ``/etc/bcfg2-web.conf``. Any database supported by `Django
   <http://www.djangoproject.com>`_ can be used.  As of version 1.3,
   `South <http://south.aeracode.org>`_ is used to control schema changes.
   If your database is not supported by South, any updates will need to
   be applied manually.  Sqlite is configured by default.  Please see the
   :ref:`reporting-databases` section to configure alternative databases.

   .. warning::

       If you are using an sqlite database, the directory containing the
       database file will need to be writable by the web server. The reason
       for this is that sqlite will create another file for its journal
       when it tries to update the database file.

   .. note::

       Distributed environments can share a single remote database for
       reporting.

2. After configuring your database be sure to run ``bcfg2-admin reports
   init`` to create the schema.

3. To enable statistics collection in the bcfg2-server, add
   :ref:`server-plugins-statistics-reporting` to the **plugins**
   line in your ``bcfg2.conf`` and restart the bcfg2-server.  A report
   collecting daemon should be run to import the collected statistics
   into the backend.  Please see the section :ref:`Report Collector
   <report_collector>` for more information.

   Detailed installation instructions can be found :ref:`here
   <appendix-guides-web-reports-install>`.

.. _dynamic-http-install:

Apache configuration for web-based reports
------------------------------------------

.. note::

    Reports no longer needs to be installed at the root URL for a given
    host. Therefore, reports no longer require their own virtual host.

    In order to make this work, you will need to specify your web prefix
    by adding a **web_prefix** setting in the [statistics] section of
    your ``bcfg2.conf``.

.. warning::

    When running with SELINUX enabled, you can have potential problems
    with the WSGISocketPrefix. One solution that works without too much
    trouble is modifying your prefix so that it is located in a standard
    location::

        WSGISocketPrefix /var/run/httpd/wsgi

An example site config is included below::

    <IfModule mod_wsgi.c>
      #
      # Read an alternate configuration file
      #
      # SetEnv BCFG2_CONFIG_FILE /etc/bcfg2_testing.conf

      #
      # If the root is changed update the static content alias as well
      #
      WSGIScriptAlias /bcfg2 "/usr/share/bcfg2/reports.wsgi"

      WSGISocketPrefix run
      WSGIDaemonProcess Bcfg2.Server.Reports processes=1 threads=10
      WSGIProcessGroup Bcfg2.Server.Reports

      #
      # Manually set this to override the static content
      #
      #SetEnv bcfg2.media_url /bcfg2/site_media/

      #
      # This should have the same prefix as WSGIScriptAlias
      #
      Alias "/bcfg2/site_media/" "/usr/share/bcfg2/site_media/"
      <Directory "/usr/share/bcfg2/site_media/">
        Options None
        AllowOverride None
        order deny,allow
        deny from all
        allow from 127.0.0.1
      </Directory>
    </IfModule>

This configuration is suitable for use with the default installation
from an RPM or deb package.

At this point you should be able to point your web browser to
http://localhost/bcfg2 and see the new reports.

Upgrading
============

1. Convert database config

    Run `tools/upgrade/1.3/migrate_configs.py`

    Beginning with 1.3 the database configuration moved from [statistics] to
    [database] in `bcfg2.conf` and `bcfg2-web.conf`.  The old settings will
    be accepted but a deprecation warning will be displayed.

2. Replace the DBStats plugin with the Reporting plugin.

3. Migrate historic data.
    Run ``tools/upgrade/1.3/migrate_dbstats.py``

    The reporting schema is now managed using `South <http://south.aeracode.org>`_
    instead of a set of custom scripts. This creates the new schema and imports
    all of the historic data to the new format.

    .. note::

        After the database is upgraded all of the old tables are left
        intact. To remove them any table starting with **reports\_** can
        be dropped.

4. `(Optional)` Run the :ref:`Report Collector <report_collector>`
    Add "transport = LocalFilesystem" under "[reporting]" in
    ``bcfg2.conf``.  Restart the bcfg2-server and start the
    bcfg2-report-collector.

Configuring
===========

Most of the configuration is handled through the ``/etc/bcfg2.conf``
or alternatively ``/etc/bcfg2-web.conf``.

An example using the defaults is listed below::

    [database]
    engine = sqlite3
    name = /var/lib/bcfg2/etc/bcfg2.sqlite
    user =
    password =
    host =
    port =

    [reporting]
    transport = DirectStore
    web_prefix =
    file_limit = 1m

Configuration Sections
----------------------

.. _reporting-databases:

database
^^^^^^^^

If you choose to use a different database, you'll need to edit
``/etc/bcfg2.conf``. These fields should be updated in the 
``[database]`` section:

* engine

  * ex: engine = mysql
  * ex: engine = postgresql_psycopg2

* name
* user
* password
* host
* port (optional)

To store reporting data separately from the main server data, use
the following options:

* reporting_engine

  * ex: reporting_engine = mysql
  * ex: reporting_engine = postgresql_psycopg2

* reporting_name
* reporting_user
* reporting_password
* reporting_host
* reporting_port (optional)

.. warning::

    If mysql is used as a backend, it is recommended to use InnoDB for
    the `storage engine <http://dev.mysql.com/doc/refman/5.1/en/storage-engine-setting.html>`_.

Refer to :ref:`server-database` for a full listing of 
available options.

statistics
^^^^^^^^^^

.. deprecated: 1.3.0

* config: The config file to be read for additional reporting
  data.  This is used to restrict what can be read by the web
  server.
* time_zone: The django TIME_ZONE settings parameter.
* web_debug: Set Django's DEBUG and TEMPLATE_DEBUG settings. This
  is known to cause memory leaks.  Use with caution!

reporting
^^^^^^^^^

* transport: See :ref:`Transports <dynamic_transports>`.
* web_prefix: Prefix to be added to Django's MEDIA_URL
* file_limit: The maximum size of a diff or binary data to
  store in the database.
* max_children: Maximum number of children for the reporting
  collector. Use 0 to disable the limit and spawn a thread
  as soon as a working file is available.


.. _dynamic_transports:

Statistics Transports
---------------------

A transport is required to pass the data collected from the bcfg2-server
to the bcfg2-report-collector.  At the time of this writing two transports
are available:

* LocalFilesystem: Statistics are written to the local file system and collected
  on the local machine.
* RedisTransport: Statistics are sent through a list in redis.
* DirectStore: DBStats style threaded imports in the main server process.

Future transports will allow multiple servers to pass data to a single or multiple
bcfg2-report-collector processes.  New installations default to and should use the
LocalFilesystem transport.  Upgrades will use DirectStore by default in the 1.3
release.

.. Note::
    If DirectStore is used, the bcfg2-report-collector process will refuse to run
    since this method is not compatible with an external process.

RedisTransport
^^^^^^^^^^^^^^

This transport uses a single redis instance for communication between bcfg2-server and
bcfg2-report-collector.  Multiple servers can write to a single redis instance and multiple
report collectors may be run as well.

An example configuration with the default values::

    [reporting]
    transport = RedisTransport
    redis_host = 127.0.0.1
    redis_port = 6379
    redis_db = 0

bcfg2-admin commands operate slightly differently in this mode.  Instead of querying the
database directly, rpc commands are issued to the report collectors.  This only affects
the minestruct and pull commands.

.. warning::

   At the time of this writing the version of python-redis in EPEL is too old to use with
   this transport.  Current versions of the python-redis package require python >= 2.5.

Usage
=====

.. _report_collector:

Report Collector daemon
-----------------------

.. Note::
    This section does not apply when the DirectStore transport is used.

The bcfg2-report-collector gathers statistics from the bcfg2-server process
and records them in the backend database.

Options are similar to the bcfg2-server daemon::

     -D <pidfile>                 Daemonize process, storing pid
     -o <path>                    Set path of file log
     -h                           Print this usage message
     -E <encoding>                Encoding of cfg files
     -W <conffile>                Web interface configuration file
     -Q <repository path>         Server repository path
     -C <conffile>                Specify configuration file
     --version                    Print the version and exit
     -d                           Enable debugging output
     -v                           Enable verbose output

.. Note::
    The bcfg2-report-collector is not set to start by default

bcfg2-admin reports (command line script)
-----------------------------------------

The bcfg2-admin tool provides management and maintenance capabilities for
the reporting database.  A few useful `Django <http://www.djangoproject.com>`_
commands are provided as well.

* init: Initialize a new database
* update: Apply any updates to the reporting database. Unlike the syncdb
  command, this will modify existing tables.
* purge: Removes unwanted clients and data.

  * -c --client [client name] - Remove interactions from a single client.
  * --expired - Remove all data for expired clients.  --days is used to exclude
    clients expired within n days.
  * --days [n] - Remove interactions older then n days.  If not used with any
    other modifiers, all data older then n days is removed.

* scrub: Scrub the database for any orphaned objects.

Django commands
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* dbshell:  Connects to the backend database.
* shell:  Starts an interactive python shell with the Django environment setup.
* sqlall:  Print the sql statements used to create the database.
* validate:  Validate the database against the current models.

bcfg2-reports (command line script)
-----------------------------------

bcfg2-reports allows you to retrieve data from the database about clients,
and the states of their current interactions. It also allows you to
change the expired/unexpired states.

The utility runs as a standalone application. It does, however, use the
models from ``/src/lib/Server/Reports/reports/models.py``.

A number of different options can be used to change what bcfg2-reports
displays::

    Usage: python bcfg2-reports [option] ...

    Options and arguments (and corresponding environment variables):
    -a                        : shows all hosts, including expired hosts
    -b NAME                   : single-host mode - shows bad entries from the
                                current interaction of NAME
    -c                        : shows only clean hosts
    -d                        : shows only dirty hosts
    -e NAME                   : single-host mode - shows extra entries from the
                                current interaction of NAME
    -h                        : shows help and usage info about bcfg2-reports
    -m NAME                   : single-host mode - shows modified entries from the
                                current interaction of NAME
    -s NAME                   : single-host mode - shows bad, modified, and extra
                                entries from the current interaction of NAME
    -t NAME                   : single-host mode - shows total number of managed and
                                good entries from the current interaction of NAME
    -x NAME                   : toggles expired/unexpired state of NAME
    --badentry=KIND,NAME      : shows only hosts whose current interaction has bad
                                entries in of KIND kind and NAME name; if a single
                                argument ARG1 is given, then KIND,NAME pairs will be
                                read from a file of name ARG1
    --modifiedentry=KIND,NAME : shows only hosts whose current interaction has
                                modified entries in of KIND kind and NAME name; if a
                                single argument ARG1 is given, then KIND,NAME pairs
                                will be read from a file of name ARG1
    --extraentry=KIND,NAME    : shows only hosts whose current interaction has extra
                                entries in of KIND kind and NAME name; if a single
                                argument ARG1 is given, then KIND,NAME pairs will be
                                read from a file of name ARG1
    --fields=ARG1,ARG2,...    : only displays the fields ARG1,ARG2,...
                                (name,time,state,total,good,bad)
    --sort=ARG1,ARG2,...      : sorts output on ARG1,ARG2,...
                                (name,time,state,total,good,bad)
    --stale                   : shows hosts which haven't run in the last 24 hours

Screenshots
===========

Grid Overview
-------------

.. image:: GridView.png
   :alt: Grid overview
   :width: 850px
   :height: 530px

Detailed Overview
-----------------

.. image:: DetailedView.png
   :alt: Detailed overview
   :width: 850px
   :height: 530px

.. _reports-calendar-summary:

Calendar Summary
----------------

.. image:: CalView.png
   :alt: Calendar summary
   :width: 850px
   :height: 530px

.. _reports-item-detail:

Client Detail
-------------

.. _reports-client-detail:

.. image:: ClientDetail.png
   :alt: Client detail
   :width: 850px
   :height: 530px

Common Problems
---------------

.. image:: CommonProblems.png
   :alt: Common configuration problems
   :width: 850px
   :height: 530px

Item Listing
------------

.. image:: BadListing.png
   :alt: Item listing
   :width: 850px
   :height: 530px

Item Detail
-----------

.. image:: ConfigItem.png
   :alt: Item detail
   :width: 850px
   :height: 530px