From 553c693618321fad2a88030b16d42d3253befaec Mon Sep 17 00:00:00 2001 From: Fabian Affolter Date: Mon, 8 Nov 2010 23:52:30 +0100 Subject: Directories moved --- doc/reports/dynamic.txt | 270 ++++++++++++++++++++++++++++++++++++++++++ doc/reports/index.txt | 29 +++++ doc/reports/install.txt | 183 ++++++++++++++++++++++++++++ doc/reports/item_detail.jpg | Bin 0 -> 98249 bytes doc/reports/node_dropdown.jpg | Bin 0 -> 148486 bytes doc/reports/static.txt | 100 ++++++++++++++++ doc/reports/summary_cal.jpg | Bin 0 -> 175328 bytes 7 files changed, 582 insertions(+) create mode 100644 doc/reports/dynamic.txt create mode 100644 doc/reports/index.txt create mode 100644 doc/reports/install.txt create mode 100644 doc/reports/item_detail.jpg create mode 100644 doc/reports/node_dropdown.jpg create mode 100644 doc/reports/static.txt create mode 100644 doc/reports/summary_cal.jpg (limited to 'doc/reports') diff --git a/doc/reports/dynamic.txt b/doc/reports/dynamic.txt new file mode 100644 index 000000000..1ec41af64 --- /dev/null +++ b/doc/reports/dynamic.txt @@ -0,0 +1,270 @@ +.. -*- mode: rst -*- + +.. _server-reports-dynamic: + +============================== +Bcfg2 Dynamic Reporting System +============================== + +.. versionadded:: 0.8.2 + +Installation +============ + +Prerequisites +------------- + +* sqlite3 +* pysqlite2 +* `Django `_ +* mod-python + +Install +------- + +Be sure to include the specified fields included in the example +``bcfg2.conf`` file. These can be specified in either ``/etc/bcfg2.conf``, +if it is readable by the webserver user, or ``/etc/bcfg2-web.conf``. Any +database supported by Django can be used. If you are not using sqlite +(the default choice), please see the :ref:`alternative-databases` +section below. + +.. note:: + + Distributed environments can share a single remote database for + reporting. + +The recommended statistics plugin is DBStats. To use it, add it to the +**plugins** line in your ``bcfg2.conf``. Alternatively, the Statistics +plugin can be used in conjunction with a crontab entry to run +``/usr/sbin/bcfg2-admin reports load_stats``. + +Detailed installation instructions can be found :ref:`here `. + +.. _dynamic-http-install: + +Apache configuration for web-based reports +------------------------------------------ + +.. note:: + + Reports no longer needs to be installed at the root URL for a given + host. Therefore, reports no longer require their own virtual host. + + In order to make this work, you will need to specify your web prefix + by adding a **web_prefix** setting in the [statistics] section of + your ``bcfg2.conf``. + +An example site config is included below for the vhost "reports.mcs.anl.gov":: + + + ServerAdmin webmaster@mcs.anl.gov + ServerName reports.mcs.anl.gov + DocumentRoot /var/www/reports + + Order deny,allow + Deny from all + Allow from localhost #you may want to change this + AllowOverride None + + + # Possible values include: debug, info, notice, warn, error, crit, + # alert, emerg. + LogLevel warn + + ServerSignature Off + + # Stop TRACE/TRACK vulnerability + + RewriteEngine on + RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) + RewriteRule .* - [F] + + + SetHandler python-program + PythonHandler django.core.handlers.modpython + SetEnv DJANGO_SETTINGS_MODULE Bcfg2.Server.Reports.settings + PythonDebug On + + + SetHandler None + + + +The first three lines of this configuration must be customized per site. + +The ``bcfg2-tarball/reports/site_media/`` directory needs to be copied +to ``/var/www/reports/site_media/`` It could live anywhere; as long as +the contents are accessible on the virtual host at ``/site_media/``. + +At this point you should be able to point your web browser to the +virtualhost you created and see the new reports + +Example WSGI configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +entry.wsgi:: + + import os, sys + os.environ['DJANGO_SETTINGS_MODULE'] = 'Bcfg2.Server.Reports.settings' + import django.core.handlers.wsgi + application = django.core.handlers.wsgi.WSGIHandler() + +Apache conf:: + + Alias /bcfg2reports/site_media "/path/to/site_media" + + Order deny,allow + Allow from all + AllowOverride None + + # If Python is installed in a non-standard prefix: + #WSGIPythonHome /python/prefix + #WSGIPythonPath /python/prefix/lib/python2.6/site-packages + WSGIScriptAlias /bcfg2reports "/another/path/to/entry.wsgi" + +.. _alternative-databases: + +Notes on Alternative Databases +------------------------------ + +If you choose to use a different database, you'll need to edit +``/etc/bcfg2.conf``. These fields should be updated in the [statistics] +section: + +* database_engine + + * ex: database_engine = mysql + * ex: database_engine = postgresql_psycopg2 + +* database_name +* database_user +* database_password +* database_host +* database_port (optional) + +Summary and Features +==================== + +The new reporting system was implemented to address a number of +deficiencies in the previous system. By storing statistics data in a +relational database, we are now able to view and analyze more information +about the state of the configuration, including information about previous +configuration. Specific features in the new system include: + +* The ability to look at a :ref:`reports-calendar-summary` with past + statistics information. +* More recent data concerning hosts. +* Additional information display in reports. Primarily, reasons for + :ref:`configuration item verification failure ` + are now accessible. +* Instead of static pages, pages are generated on the fly, allowing + users to drill down to find out about a :ref:`specific host + `, rather than only having one huge page with + too much information. + +Planned improvements include +============================ + +* Accounts, customized displays for each admin. And privacy of config data. +* Config browsing capabilities; to look at your config in an interesting way. +* Fixing all the known bugs from below. + +Unfortunately with all the improvements, there are a few less exciting +elements about the new reporting system. The new reporting system +moves away from static pages and towards a real web application, which +causes mainly problems with dependencies and makes installation more +difficult. This should become less of a problem over time as we develop +a better installation process for a web application. + +Usage +===== + +bcfg2-admin reports (command line script) +----------------------------------------- + +The bcfg2-admin tool provides management and maintenance capabilities for +the reporting database. A few useful `Django `_ +commands are provided as well. + +* init: Initialize a new database +* load_stats: Load statistics data from the Statistics plugin into the + database. This was importscript.py. +* scrub: Scrub the database for duplicate reasons. +* update: Apply any updates to the reporting database. Unlike the syncdb + command, this will modify existing tables. + +Django commands +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +* syncdb: Create the tables for any models not installed. Django will + not modify any existing tables. +* sqlall: Print the sql statements used to create the database. Note: + This does not show the fixture data. +* validate: Validate the database against the current models. + +bcfg2-reports (command line script) +----------------------------------- + +bcfg2-reports allows you to retrieve data from the database about clients, +and the states of their current interactions. It also allows you to +change the expired/unexpired states. + +The utility runs as a standalone application. It does, however, use the +models from ``/src/lib/Server/Reports/reports/models.py``. + +A number of different options can be used to change what bcfg2-reports +displays:: + + Usage: python bcfg2-reports [option] ... + + Options and arguments (and corresponding environment variables): + -a : shows all hosts, including expired hosts + -b NAME : single-host mode - shows bad entries from the + current interaction of NAME + -c : shows only clean hosts + -d : shows only dirty hosts + -e NAME : single-host mode - shows extra entries from the + current interaction of NAME + -h : shows help and usage info about bcfg2-reports + -s NAME : single-host mode - shows bad and extra entries from + the current interaction of NAME + -x NAME : toggles expired/unexpired state of NAME + --badentry=KIND,NAME : shows only hosts whose current interaction has bad + entries in of KIND kind and NAME name; if a single + argument ARG1 is given, then KIND,NAME pairs will be + read from a file of name ARG1 + --extraentry=KIND,NAME : shows only hosts whose current interaction has extra + entries in of KIND kind and NAME name; if a single + argument ARG1 is given, then KIND,NAME pairs will be + read from a file of name ARG1 + --fields=ARG1,ARG2,... : only displays the fields ARG1,ARG2,... + (name,time,state) + --sort=ARG1,ARG2,... : sorts output on ARG1,ARG2,... (name,time,state) + --stale : shows hosts which haven't run in the last 24 hours + +Screenshots +=========== + +.. _reports-calendar-summary: + +Calendar Summary +---------------- + +.. image:: summary_cal.jpg + :alt: Calendar summary + +.. _reports-item-detail: + +Item detail +----------- + +.. image:: item_detail.jpg + :alt: Item detail + +Node dropdown +------------- + +.. _reports-node-dropdown: + +.. image:: node_dropdown.jpg + :alt: Node dropdown diff --git a/doc/reports/index.txt b/doc/reports/index.txt new file mode 100644 index 000000000..bef686305 --- /dev/null +++ b/doc/reports/index.txt @@ -0,0 +1,29 @@ +.. -*- mode: rst -*- + +.. _server-reports-index: + +The Bcfg2 Reporting System +========================== + +Bcfg2's reporting system is its killer feature. It allows administrators +to gain a broad understanding of the configuration state of their entire +environment. It summarizes + +* Configuration changes and when they were made +* Discrepancies between the specification and current client states + + * Clients can be grouped by misconfiguration type + +* Configuration entries that are not specified +* Overall client summaries according to these types + +There are two systems, the old system, which builds static reports based +on a series of XSLT stylesheets and a new dynamic reporting system that +uses django and a database backend. + +.. toctree:: + :maxdepth: 2 + + install + static + dynamic diff --git a/doc/reports/install.txt b/doc/reports/install.txt new file mode 100644 index 000000000..80f9342ae --- /dev/null +++ b/doc/reports/install.txt @@ -0,0 +1,183 @@ +.. -*- mode: rst -*- + +.. _EPEL: http://fedoraproject.org/wiki/EPEL + +.. This is combination of the Ubuntu guide and the Centos guide for + installing the web reports. + +.. _server-reports-install: + +===================== +Dynamic (web) Reports +===================== + +The first step is to install the needed software components like the +Django framework and the database (SQlite2). All packages for Fedora +are in the Fedora Package Collection or in EPEL_ for CentOS/RHEL:: + + [root@system01 ~]# yum -y install Django python-simplejson python-sqlite2 + +Of course is a web server needed as well:: + + [root@system01 ~]# yum -y install httpd mod_python + +The same packages are needed for Ubuntu systems:: + + [root@system01 ~]# aptitude install python-django apache2 libapache2-mod-python + +Now we need to create the sqlite database. Use the following command on +Fedora, CentOS, or RHEL.:: + + [root@system01 ~]# python /usr/lib/python2.4/site-packages/Bcfg2/Server/Reports/manage.py syncdb + Creating table auth_permission + Creating table auth_group + Creating table auth_user + Creating table auth_message + Creating table django_content_type + Creating table django_session + Creating table django_site + Creating table django_admin_log + Creating table reports_client + Creating table reports_ping + Creating table reports_interaction + Creating table reports_reason + Creating table reports_entries + Creating table reports_entries_interactions + Creating table reports_performance + Creating table reports_internaldatabaseversion + + You just installed Django's auth system, which means you don't have any superusers defined. + Would you like to create one now? (yes/no): no + Installing index for auth.Permission model + Installing index for auth.Message model + Installing index for admin.LogEntry model + Installing index for reports.Client model + Installing index for reports.Ping model + Installing index for reports.Interaction model + Installing index for reports.Entries model + Installing index for reports.Entries_interactions model + +.. note:: There are different versions of Python available. If you are + unsure about your installed version use the following line instead of + the line above.:: + + [root@system01 ~]# PYVER=`python -c 'import sys;print(sys.version[0:3])'`; python /usr/lib/python$PYVER/site-packages/Bcfg2/site-packages/Bcfg2/Server/Reports/manage.py syncdb + +The path on Ubuntu systems is different. Please use the same path as shown +in the following command to execute the script on an Ubuntu machine in +the next steps:: + + [root@system01 ~]# python /usr/share/pyshared/Bcfg2/Server/Reports/manage.py syncdb + Creating table auth_permission + Creating table auth_group + Creating table auth_user + Creating table auth_message + Creating table django_content_type + Creating table django_session + Creating table django_site + Creating table django_admin_log + Creating table reports_client + Creating table reports_ping + Creating table reports_interaction + Creating table reports_reason + Creating table reports_entries + Creating table reports_entries_interactions + Creating table reports_performance + Creating table reports_internaldatabaseversion + + You just installed Django's auth system, which means you don't have any superusers defined. + Would you like to create one now? (yes/no): no + Installing index for auth.Permission model + Installing index for auth.Message model + Installing index for admin.LogEntry model + Installing index for reports.Client model + Installing index for reports.Ping model + Installing index for reports.Interaction model + Installing index for reports.Entries model + Installing index for reports.Entries_interactions model + +The server should be tested to make sure that there are no mistakes:: + + [root@system01 ~]# python /usr/lib/python2.6/site-packages/Bcfg2/Server/Reports/manage.py testserver + Creating test database... + Creating table auth_permission + Creating table auth_group + Creating table auth_user + Creating table auth_message + Creating table django_content_type + Creating table django_session + Creating table django_site + Creating table django_admin_log + Creating table reports_client + Creating table reports_ping + Creating table reports_interaction + Creating table reports_reason + Creating table reports_entries + Creating table reports_entries_interactions + Creating table reports_performance + Creating table reports_internaldatabaseversion + Installing index for auth.Permission model + Installing index for auth.Message model + Installing index for admin.LogEntry model + Installing index for reports.Client model + Installing index for reports.Ping model + Installing index for reports.Interaction model + Installing index for reports.Entries model + Installing index for reports.Entries_interactions model + Validating models... + 0 errors found + + Django version 1.1.1, using settings 'Reports.settings' + Development server is running at http://127.0.0.1:8000/ + Quit the server with CONTROL-C. + +Add DBStats to the plugins line of ``bcfg2.conf``. The resulting +**[server]** section should look something like this:: + + [server] + repository = /var/lib/bcfg2 + plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase + +Start/restart the Bcfg2 server:: + + [root@system01 ~]# /etc/init.d/bcfg2-server restart + +Run the Bcfg2 client in order to populate the statistics database +(this run should take a bit longer since you are uploading the client +statistics to the database). + +Download the static reports content:: + + [root@system01 ~]# cd /var/www/ + [root@system01 ~]# svn co https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2/reports + +Configure Apache using :ref:`dynamic-http-install` as a guide + +Copy server/statistics sections of ``bcfg2.conf`` to +``/etc/bcfg2-web.conf`` (make sure it is world-readable). You should +then have something like this:: + + [server] + repository = /var/lib/bcfg2 + plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase + + [statistics] + sendmailpath = /usr/lib/sendmail + database_engine = sqlite3 + # 'postgresql', 'mysql', 'mysql_old', 'sqlite3' or 'ado_mssql'. + database_name = + # Or path to database file if using sqlite3. + #/etc/brpt.sqlite is default path if left empty + database_user = + # Not used with sqlite3. + database_password = + # Not used with sqlite3. + database_host = + # Not used with sqlite3. + database_port = + # Set to empty string for default. Not used with sqlite3. + web_debug = True + +Restart apache and point a browser to your Bcfg2 server. + +If using sqlite be sure the sql database file and directory containing the database are writable to apache. diff --git a/doc/reports/item_detail.jpg b/doc/reports/item_detail.jpg new file mode 100644 index 000000000..198880599 Binary files /dev/null and b/doc/reports/item_detail.jpg differ diff --git a/doc/reports/node_dropdown.jpg b/doc/reports/node_dropdown.jpg new file mode 100644 index 000000000..d3f7d4f14 Binary files /dev/null and b/doc/reports/node_dropdown.jpg differ diff --git a/doc/reports/static.txt b/doc/reports/static.txt new file mode 100644 index 000000000..d5d96fe12 --- /dev/null +++ b/doc/reports/static.txt @@ -0,0 +1,100 @@ +.. -*- mode: rst -*- + +.. _server-reports-static: + +============================= +Bcfg2 Static Reporting System +============================= + +The Bcfg2 reporting system collects and displays information about the +operation of the Bcfg2 client, and the configuration states of target +machines. + +Goals +===== + +The reporting system provides an interface to administrators describing +a few important tasks + +* Client configuration state, particularly aspects that do not match the configuration specification. + Information about bad and extra configuration elements is included. +* Client execution results (a list of configuration elements that were modified) +* Client execution performance data (including operation retry counts, and timings for several critical execution regions) + +This data can be used to understand the current configuration state +of the entire network, the operations performed by the client, how the +configuration changes propagate, and any reconfiguration operations that +have failed. + +Retention Model +=============== + +The current reporting system stores statistics in an XML data store, by +default to {{{/etc/statistics.xml}}}. It retains either one or two +statistic sets per host. If the client has a clean configuration state, +the most recent (clean) record is retained. If the client has a dirty +configuration state, two records are retained. One record is the last +clean record. The other record is the most recent record collected, +detailing the incorrect state. + +This retention model, while non-optimal, does manage to persistently +record most of the data that users would like. + +Setup +===== + +In order to configure your Bcfg2 server for receiving reports, you +will need to list the Statistics plugin in the plugins line of your +``bcfg2.conf``. You will also need a [statistics] section +in your ``bcfg2.conf``. You can find out more about what goes there in the +``bcfg2.conf`` manpage. + +Output +====== + +Several output reports can be generated from the statistics store with +the command line tool {{{bcfg2-build-reports}}}. + +* Nodes Digest +* Nodes Individual +* Overview Statistics +* Performance + +The data generated by these reports can be delivered by several +mechanisms: + +* HTML +* Email +* RSS + +Shortcomings and Planned Enhancements +===================================== + +When designing the current reporting system, we were overly concerned with +the potential explosion in data size over time. In order to address this, +we opted to use the retention scheme described above. This approach has +several shortcomings: + +* A comprehensive list of reconfiguration operations (with associated + timestamps) isn't retained +* Client results for any given day (except the last one) aren't uniformly + retained. This means that inter-client analysis is difficult, if + not impossible + +We plan to move to a database backend to address the dataset size +problem and start retaining all information. The move to a SQL backend +will allow many more types of queries to be efficiently processed. It +will also make on-demand reports simpler. + +Other sorts of information would also be useful to track. We plan to +add the ability to tag a particular configuration element as security +related, and include this in reports. This will aid in the effective +prioritization of manual and failed reconfiguration tasks. + +Capability Goals (posed as questions) +------------------------------------- + +* What machines have not yet applied critical updates? +* How long did critical updates take to be applied? +* What configuration did machine X have on a particular date? +* When did machine X perform configuration update Y? diff --git a/doc/reports/summary_cal.jpg b/doc/reports/summary_cal.jpg new file mode 100644 index 000000000..fab010762 Binary files /dev/null and b/doc/reports/summary_cal.jpg differ -- cgit v1.2.3-1-g7c22