diff options
| author | Sol Jerome <solj@ices.utexas.edu> | 2009-12-29 04:19:02 +0000 |
|---|---|---|
| committer | Sol Jerome <solj@ices.utexas.edu> | 2009-12-29 04:19:02 +0000 |
| commit | bd0204ecb1fb80cdf36af0f57b72e84445c1a088 (patch) | |
| tree | 930345633221d9fda156140fb494d739a0484f85 /doc | |
| parent | d61a93ac7451be4eedb07f93d507b67d6af7b025 (diff) | |
| download | bcfg2-bd0204ecb1fb80cdf36af0f57b72e84445c1a088.tar.gz bcfg2-bd0204ecb1fb80cdf36af0f57b72e84445c1a088.tar.bz2 bcfg2-bd0204ecb1fb80cdf36af0f57b72e84445c1a088.zip | |
doc: Rearrange plugin document structure
Signed-off-by: Sol Jerome <solj@ices.utexas.edu>
git-svn-id: https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2@5636 ce84e21b-d406-0410-9b95-82705330c041
Diffstat (limited to 'doc')
36 files changed, 1469 insertions, 86 deletions
diff --git a/doc/index.txt b/doc/index.txt index 2761c3270..05baefd22 100644 --- a/doc/index.txt +++ b/doc/index.txt @@ -16,8 +16,6 @@ Welcome to Bcfg2's documentation! unsorted/index - .. FIXME: Platform-specific pages would be good here - glossary Indices and tables diff --git a/doc/plugins/account.txt b/doc/plugins/account.txt deleted file mode 100644 index 311d91b24..000000000 --- a/doc/plugins/account.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -======= -Account -======= diff --git a/doc/plugins/cfg.txt b/doc/plugins/cfg.txt deleted file mode 100644 index 5701f2327..000000000 --- a/doc/plugins/cfg.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -=== -Cfg -=== diff --git a/doc/plugins/decisions.txt b/doc/plugins/decisions.txt deleted file mode 100644 index 3e32bd803..000000000 --- a/doc/plugins/decisions.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -========= -Decisions -========= diff --git a/doc/plugins/deps.txt b/doc/plugins/deps.txt deleted file mode 100644 index 86b3a272e..000000000 --- a/doc/plugins/deps.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -==== -Deps -==== diff --git a/doc/plugins/generators/account.txt b/doc/plugins/generators/account.txt new file mode 100644 index 000000000..e07cef8b6 --- /dev/null +++ b/doc/plugins/generators/account.txt @@ -0,0 +1,24 @@ +.. -*- mode: rst -*- + +======= +Account +======= + +The account plugin manages authentication data, including + +* /etc/passwd +* /etc/group +* /etc/security/limits.conf +* /etc/sudoers +* /root/.ssh/authorized_keys + +User access data is stored in three files in the Account directory: + +* superusers (a list of users who always have root privs) +* rootlist (a list of user:host pairs for scoped root privs) +* useraccess (a list of user:host pairs for login access) + +SSH keys are stored in files named $username.key; these are installed into root's authorized keys for users in the superusers list as well as for the pertitent users in the rootlike file (for the current system). + +Authentication data is read in from (static|dyn).(passwd|group) The static ones are for system local ones, while the dyn. versions are for external synchronization (from ldap/nis/etc) +There is also a static.limits.conf that provides the limits.conf header and any static entries. diff --git a/doc/plugins/generators/cfg.txt b/doc/plugins/generators/cfg.txt new file mode 100644 index 000000000..8ef9b17dc --- /dev/null +++ b/doc/plugins/generators/cfg.txt @@ -0,0 +1,103 @@ +.. -*- mode: rst -*- + +=== +Cfg +=== + +The Cfg plugin provides a repository to describe configuration file contents for clients. In its simplest form, the Cfg repository is just a directory tree modeled off of the directory tree on your client machines. + +The Cfg Repository +================== + +The Cfg plugin is enabled by including `Cfg` on the `plugins` line of the `[server]` section of your Bcfg2 server config file. The repository itself lives in `/var/lib/bcfg2/Cfg`, assuming you are using the default repository location of `/var/lib/bcfg2`. The contents of this directory are a series of directories corresponding to the real-life locations of the files on your clients, starting at the root level. For example:: + + lueningh@tg-prez:~/bcfg2/repository> ls Cfg + bin/ boot/ etc/ opt/ root/ usr/ var/ + +Specific config files go in like-named directories in this heirarchy. For example the password file, `/etc/passwd`, goes in `Cfg/etc/passwd/passwd`, while the ssh pam module config file, `/etc/pam.d/sshd`, goes in `Cfg/etc/pam.d/sshd/sshd`. The reason for the like-name directory is to allow multiple versions of each file to exist, as described below. Note that these files are exact copies of what will appear on the client machine - no templates, XML wrappers, etc. + +Group-Specific Files +==================== + +It is often that you want one version of a config file for all of your machines except those in a particular group. For example, `/etc/fstab` should look alike on all of your desktop machines, but should be different on your file servers. Bcfg can handle this case through use of group-specific files. + +As mentioned above, all Cfg entries live in like-named directories at the end of their directory tree. In the case of `fstab`, the file at `Cfg/etc/fstab/fstab` will be handed out by default to any client that asks for a copy of `/etc/fstab`. Group-specific files are located in the same directory and are named with the syntax:: + + /path/to/filename/filename.GNN_groupname + +in which ''NN'' is a priority number where '00' is lowest and '99' is highest, and ''groupname'' is the name of a group defined in `Metadata/groups.xml`. Back to our `fstab` example, we might have a `Cfg/etc/fstab/` directory that looks like:: + + fstab + fstab.G50_server + fstab.G99_fileserver + +By default, clients will receive the plain `fstab` file when they request `/etc/fstab`. Any machine that is in the `server` group, however, will instead receive the `fstab.G50_server` file. Finally, any machine that is in the `fileserver` group will receive the `fstab.G99_fileserver` file, even if they are also in the `server` group. + +Host-Specific Files +=================== + +Similar to the case with group-specific files, there are cases where a specific machine should have a different version of a file than all others. This can be accomplished with host-specific files. The format of a host-specific file name is:: + + /path/to/filename/filename.H_host.example.com + +Host-specific files have a higher priority than group specific files. Again, the `fstab` example:: + + fstab + fstab.G50_server + fstab.G99_fileserver + fstab.H_host.example.com + +In this case, `host.example.com` will always get the host-specific version, even if it is part of the `server` or `fileserver` (or both) classes. + +.. note:: If you have the ability to choose between using a group-specific file and a host-specific file, it is almost always best to use a group-specific one. That way if a hostname changes or an extra copy of a particular client is built, it will get the same changes as the original. + +Info files +========== + +By default, Cfg writes files to the filesystem with owner `root`, group `root`, and mode 644 (read and write for owner, read only for group and other). These options, and a few others, can be overridden through use of `:info` files. Each config file directory can have a `:info` file if needed. The possible fields in a `:info` file are: + ++-----------+-------------------+------------------------------------------------------+---------+ +| Field | Possible values | Description | Default | ++===========+===================+======================================================+=========+ +| owner: | Any valid user | Sets owner of the file | root | ++-----------+-------------------+------------------------------------------------------+---------+ +| group: | Any valid group | Sets group of the file | root | ++-----------+-------------------+------------------------------------------------------+---------+ +| perms: | Numeric file mode | Sets the permissions of the file | 0644 | ++-----------+-------------------+------------------------------------------------------+---------+ +| encoding: | ascii | base64 | Encoding of the file. Use base64 for non-ASCII files | ascii | ++-----------+-------------------+------------------------------------------------------+---------+ +| paranoid: | yes | no | Backup file before replacement? | no | ++-----------+-------------------+------------------------------------------------------+---------+ + +A sample `:info` file for CGI script on a web server might look like:: + + owner: www + group: www + perms: 0755 + +Back to the `fstab` example again, our final `Cfg/etc/fstab/` directory might look like:: + + :info + fstab + fstab.G50_server + fstab.G99_fileserver + fstab.H_host.example.com + +info.xml files +============== + +This feature is included in version 0.9.5pre3 and newer of the bcfg2 server. + +info.xml files add the ability to specify different sets of file metadata on a group by group basis. These files are XML, and work similarly to those used by [wiki:Plugins/Rules Rules] or [wiki:Plugins/Pkgmgr Pkgmgr]. + +The following specifies a different global set of permissions (root/sys/0651) than on clients in group webserver (root/root/0652) + +.. code-block:: xml + + <FileInfo> + <Group name='webserver'> + <Info owner='root' group='root' perms='0652'/> + </Group> + <Info owner='root' group='sys' perms='0651'/> + </FileInfo> diff --git a/doc/plugins/generators/decisions.txt b/doc/plugins/generators/decisions.txt new file mode 100644 index 000000000..17bc5c66c --- /dev/null +++ b/doc/plugins/generators/decisions.txt @@ -0,0 +1,29 @@ +.. -*- mode: rst -*- + +========= +Decisions +========= + +This page describes the Decisions plugin. The client has support for a centralized set of per-entry installation decisions. This approach is needed when particular changes are deemed "high risk"; this gives the ability to centrally specify these changes, but only install them on clients when administrator supervision is available. Because collaborative configuration is one of the remaining hard issues in configuration management, these issues typically crop up in environments with several administrators and much configuration variety. + +In these cases, the client can be configured to run in either a whitelist or blacklist mode, wherein a list of entries is downloaded from the server. The client uses this list to determine which incorrect entries should be corrected during the current run of the installation tool. The Decision plugin is the only stock plugin that generates entries for client's whitelists or blacklists. + +The Decision plugin uses a directory in the bcfg2 repository called Decisions. Files in the Decisions subdirectory are named similarly to files managed by Cfg, probes, TCheetah, and TGenshi. File basenames are either whitelist or blacklist. These files have a simple format; the following is an example. + +.. code-block:: xml + + $ cat Decisions/whitelist + <Decisions> + <Decision type='Service' name='*'/> + <Decision type='Path' name='/etc/apt/apt.conf'/> + </Decisions> + +.. note:: To add syntax highlighting in vim, you can add a modeline such as this: + + <!-- + vim: ft=xml + --> + +This example, included as a whitelist due to its name, enables all services, and the path entry named /etc/apt/apt.conf to be installed on systems running in whitelist mode; all other entry installation will be surpressed. + +When a client askes for its whitelist or blacklist, all of the files pertaining to that client of the correct type are aggregated into a single list. This list is sent to the client. Note that this list is only generated when a client is run with the appropriate option (-l (whitelist|blacklist)); client behavior is not controlled unless this option is used. diff --git a/doc/plugins/generators/deps.txt b/doc/plugins/generators/deps.txt new file mode 100644 index 000000000..9b2772f72 --- /dev/null +++ b/doc/plugins/generators/deps.txt @@ -0,0 +1,48 @@ +.. -*- mode: rst -*- + +==== +Deps +==== + +The Deps Plugin allows you to make a series of assertions like "Package X requires Package Y (and optionally also Package Z etc.) +Note that only configuration entries, like Package, !ConfigFile, etc can be used. Groupings (like Bundle) are not supported. + +Here are some examples: + +Deps/bcfg2.xml +============== + +.. code-block:: xml + + <Dependencies priority='0'> + <Package name='bcfg2'> + <Package name='python-lxml'/> + <Package name='isprelink'/> + </Package> + </Dependencies> + +This basically causes any configuration specification that includes Package bcfg2 to include python-lxml and isprelink, in a second base clause. + +Deps/bcfg2-server.xml +===================== + +.. code-block:: xml + + <Dependencies priority='0'> + <Package name='bcfg2-server'> + <Package name='python-cheetah'/> + <Package name='gamin-python'/> + <Package name='sqlite'/> + <Package name='python-sqlite'/> + <Package name='Django'/> + <Package name='mod_python'/> + <Package name='graphviz'/> + <Package name='xorg-x11-font-utils'/> + <Package name='chkfontpath'/> + <Package name='ttmkfdir'/> + <Package name='xorg-x11-xfs'/> + <Package name='urw-fonts'/> + </Package> + </Dependencies> + +This states that the bcfg2-server package (it's a separate package on some distros) depends on a long list of other packages. diff --git a/doc/plugins/generators/hostbase.txt b/doc/plugins/generators/hostbase.txt new file mode 100644 index 000000000..4b85fecd0 --- /dev/null +++ b/doc/plugins/generators/hostbase.txt @@ -0,0 +1,174 @@ +.. -*- mode: rst -*- + +======== +Hostbase +======== + +IP management system built on top of Bcfg2. It has four main parts: a django data model, a web frontend, command-line utilities, and a Bcfg2 plugin that generates dhcp, dns, and yp configuration files. + +Installation +============ + +Overview +-------- + +Installation of Hostbase requires installation of a python module, configuration of database (mysql or postgres), and configuration of an Apache webserver with mod_python. Hostbase was developed using MySQL, so this document is aimed at MySQL users. + +Prerequisites +------------- + + * mysql + * python-mysqldb + * [http://www.djangoproject.com Django] 0.95 or greater + +Install +------- + +'''Install Hostbase python module:''' + +As of Bcfg2 v0.8.7; the Hostbase module has been renamed Bcfg2.Server.Hostbase, and is automatically installed; there is nothing to do in this step. + +'''Configure the database''' + +Create the hostbase database and a user. For MySQL users:: + + mysql> CREATE DATABASE hostbase + mysql> quit + + systemprompt#: mysql -u root hostbase + mysql> GRANT ALL PRIVILEGES ON *.* TO hostbaseuser@mycomputer.private.net IDENTIFIED + BY 'password' WITH GRANT OPTION; + mysql> quit + +As of Bcfg2 v0.8.7 configuration options for Hostbase have moved to {{{/etc/bcfg2.conf}}}. There is an example bcfg2.conf with Hostbase options located at {{{bcfg2-tarball/examples/bcfg2.confHostbase}}}. Edit the hostbase options to correspond to the database you've initialized and copy the configuration to {{{/etc/bcfg2.conf}}}. To finish creating the database, from your {{{path to python/Bcfg2/Server/Hostbase}}} directory, run {{{python manage.py syncdb}}} to do all table creation. + +'''Configure the web interface''' + +Now it's possible to explore the Hostbase web interface. For curiosity, you can run Django's built-in development server to take a peek. Do this by running {{{python manage.py runserver [servername:port]}}} from your Hostbase directory. Django will default to {{{localhost:8000}}} if no server or port is entered. Now you can explore the web interface. Try adding a host and a zone. You'll see that a ".rev" zone already exists. This is where information for reverse files will go. + +For production, you'll want to have this configured for Apache with mod_python. Here is an example of how to configure Hostbase as a virtual host. + +.. code-block:: html + + <VirtualHost hostbase.mcs.anl.gov:80> + ServerAdmin systems@mcs.anl.gov + + DocumentRoot /var/www/hostbase/ + <Directory /> + AllowOverride None + </Directory> + + # Possible values include: debug, info, notice, warn, error, crit, + # alert, emerg. + LogLevel warn + + ServerSignature Off + + # Stop TRACE/TRACK vulnerability + <IfModule mod_rewrite.c> + RewriteEngine on + RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) + RewriteRule .* - [F] + </IfModule> + + Redirect / https://hostbase.mcs.anl.gov/ + </VirtualHost> + + <VirtualHost hostbase.mcs.anl.gov:443> + ServerAdmin systems@mcs.anl.gov + + DocumentRoot /var/www/hostbase/ + <Directory /> + AllowOverride None + </Directory> + + # Possible values include: debug, info, notice, warn, error, crit, + # alert, emerg. + LogLevel warn + + ServerSignature Off + + # Stop TRACE/TRACK vulnerability + <IfModule mod_rewrite.c> + RewriteEngine on + RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) + RewriteRule .* - [F] + </IfModule> + + SSLEngine On + SSLCertificateFile /etc/apache2/ssl/hostbase_server.crt + SSLCertificateKeyfile /etc/apache2/ssl/hostbase_server.key + + <Location "/"> + SetHandler python-program + PythonHandler django.core.handlers.modpython + SetEnv DJANGO_SETTINGS_MODULE Bcfg2.Server.Hostbase.settings + PythonDebug On + </Location> + <Location "/site_media/"> + SetHandler None + </Location> + </VirtualHost> + + +You'll need to copy the contents of {{{Hostbase/media}}} into {{{/var/www/hostbase/site_media}}} in this configuration to serve the correct css files. + +'''Enable the Hostbase plugin''' + +Now that the database is accessible and there is some data in it, you can enable the Hostbase plugin on your bcfg2 server to start generating some configuration files. All that needs to be done is to add {{{Hostbase}}} to the end of the list of generators in your bcfg2.conf file. To see what's being generated by Hostbase, fire up a bcfg2 development server: {{{bcfg2-info}}}. For more information on how to use the bcfg2 development server, type help at the prompt. For our purposes, type {{{debug}}}. This will bring you to an interactive python prompt where you can access bcfg's core data. + +.. code-block:: python + + for each in bcore.plugins['Hostbase'].filedata: + print each + + +The above loop will print out the name of each file that was generated by Hostbase. You can see the contents of any of these by typing {{{print bcore.plugins['Hostbase'].filedata[filename]}}}. + +'''Create a bundle''' + +Bcfg2 needs a way to distribute the files generated by Hostbase. We'll do this with a bundle. In bcfg's {{{Bundler}}} directory, touch {{{hostbase.xml}}}. + +.. code-block:: xml + + <Bundle name='hostbase' version='0.1'> + <Package name='dhcp3-server'/> + <Package name='bind9'/> + <Service name='dhcp3-server'/> + <Service name='bind9'/> + <ConfigFile name='/etc/dhcp3/dhcpd.conf'/> + <ConfigFile name='/etc/bind/[your domain]'/> + <ConfigFile name='/etc/bind/xxx.xxx.xxx.rev'/> + </Bundle> + +The above example is a bundle that will deliver both dhcp and dns files. This can be trivially split into separate bundles. It is planned that Hostbase will eventually be able to generate the list of {{{ConfigFiles}}} in its bundles automatically. + + +'''Do a Hostbase push''' + +You'll want to be able to trigger the Hostbase plugin to rebuild it's config files and push them out when data has been modified in the database. This can be done through and XMLRPC function available from the Bcfg2 server. From a client that is configured to receive one or more hostbase bundles, you'll need to first edit your {{{python/site-packages/Bcfg2/Client/Proxy.py}}} file. Add {{{'Hostbase.rebuildState'}}} to the list of methods in the bcfg2 client proxy object. The modified list is shown below: + +.. code-block:: python + + class bcfg2(ComponentProxy): + '''bcfg2 client code''' + name = 'bcfg2' + methods = ['AssertProfile', 'GetConfig', 'GetProbes', 'RecvProbeData', 'RecvStats', 'Hostbase.rebuildState'] + +Now copy the file {{{hostbasepush.py}}} from {{{bcfg2/tools}}} in the bcfg2 source to your machine. When this command is run as root, it triggers the Hostbase to rebuild it's files, then runs the bcfg2 client on your local machine to grab the new configs. + + +Authentication +============== + +Edit Django settings +-------------------- + +Django allows for custom authentication backends to its login procedure. Hostbase has an NIS authentication backend that verifies a user to be in the unix group allowed to modify Hostbase. + +To enable this feature: + * first edit your {{{Hostbase/settings.py}}} file and uncomment the line {{{'Hostbase.backends.NISBackend',}}} in the list of {{{AUTHENTICATION_BACKENDS}}} + * enter the name of the unix group you want to give access to Hostbase in the {{{AUTHORIZED_GROUP}}} variable + * in your {{{Hostbase/hostbase/views.py}}} file at the very bottom, uncomment the block(s) of lines that give you the desired level of access + +Hostbase will now direct the user to a login page if he or she is not authorized to view a certain page. Users should log in with their regular Unix username and password. diff --git a/doc/plugins/generators/nagiosgen.txt b/doc/plugins/generators/nagiosgen.txt new file mode 100644 index 000000000..55b6063a5 --- /dev/null +++ b/doc/plugins/generators/nagiosgen.txt @@ -0,0 +1,164 @@ +.. -*- mode: rst -*- + +========= +NagiosGen +========= + +This page describes the installation and use of the [http://trac.mcs.anl.gov/projects/bcfg2/browser/trunk/bcfg2/src/lib/Server/Plugins/NagiosGen.py NagiosGen] plugin. + +Update /etc/bcfg2.conf, adding NagiosGen to plugins:: + + plugins = SSHbase,Cfg,Pkgmgr,Rules,TCheetah,TWbase,NagiosGen + +Create the NagiosGen directory:: + + $ mkdir /var/lib/bcfg2/NagiosGen + +Create default host, and group specs in: + +* /var/lib/bcfg2/NagiosGen/default-host.cfg:: + + define host{ + name default + check_command check-host-alive + check_interval 5 + check_period 24x7 + contact_groups admins + event_handler_enabled 1 + failure_prediction_enabled 1 + flap_detection_enabled 1 + initial_state o + max_check_attempts 10 + notification_interval 0 + notification_options d,u,r + notification_period workhours + notifications_enabled 1 + process_perf_data 0 + register 0 + retain_nonstatus_information 1 + retain_status_information 1 + retry_interval 1 + } + +* /var/lib/bcfg2/NagiosGen/default-group.cfg:: + + define service{ + name default-service + active_checks_enabled 1 + passive_checks_enabled 1 + obsess_over_service 0 + check_freshness 0 + notifications_enabled 1 + event_handler_enabled 1 + flap_detection_enabled 1 + process_perf_data 0 + retain_status_information 1 + retain_nonstatus_information 1 + is_volatile 0 + + check_period 24x7 + max_check_attempts 4 + check_interval 5 + retry_interval 1 + contact_groups admins + notification_options w,u,c,r + notification_interval 0 + notification_period workhours + } + +Create group configuration files (Named identical to Bcfg2 groups) and +add services, and commands specific to the hostgroup (Bcfg2 group) in + +* /var/lib/bcfg2/NagiosGen/base-group.cfg:: + + define hostgroup{ + hostgroup_name base + alias base + notes Notes + } + + define service{ + service_description NTP + check_command check_ntp! + use default-service + hostgroup_name base + } + + define command{ + command_name check_ssh + command_line $USER1$/check_ssh $ARG1$ $HOSTADDRESS$ + } + + define service{ + service_description SSH + check_command check_ssh! + use default-service + hostgroup_name base + } + +* /var/lib/bcfg2/NagiosGen/web-server-group.cfg:: + + define hostgroup{ + hostgroup_name web-server + alias Port 80 Web Servers + notes UC/ANL Teragrid Web Servers Running on Port 80 + } + + define command{ + command_name check_http_80 + command_line $USER1$/check_http $HOSTADDRESS$ + } + + define service{ + service_description HTTP:80 + check_command check_http_80! + use default-service + hostgroup_name web-server + } + +Create a nagios bcfg2 bundle /var/lib/bcfg2/Bundler/nagios.xml + +.. code-block:: xml + + <Bundle name='nagios' version='2.0'> + <ConfigFile name='/etc/nagiosgen.status'/> + <Group name='rh'> + <Group name='nagios-server'> + <ConfigFile name='/etc/nagios/nagiosgen.cfg'/> + <Package name='libtool-libs'/> + <Package name='nagios'/> + <Package name='nagios-www'/> + <Service name='nagios'/> + </Group> + </Group> + <Group name='debian-lenny'> + <Group name='nagios-server'> + <ConfigFile name='/etc/nagios3/nagiosgen.cfg' + altsrc='/etc/nagios/nagiosgen.cfg'/> + <Package name='nagios3'/> + <Package name='nagios3-common'/> + <Package name='nagios3-doc'/> + <Package name='nagios-images'/> + <Service name='nagios3'/> + </Group> + </Group> + </Bundle> + +Assign clients to nagios groups in +/var/lib/bcfg2/Metadata/groups.xml + +.. code-block:: xml + + <Group name='nagios'> + <Bundle name='nagios'/> + </Group> + + <Group name='nagios-server'> + <Bundle name='nagios'/> + </Group> + +Update nagios configuration file to use nagiosgen.cfg:: + + cfg_file=/etc/nagios/nagiosgen.cfg + +Note that some of these files are built on demand, each time a client in group "nagios-server" checks in with the bcfg2 server. Local nagios instances can be configured to use the !NagiosGen directory in the bcfg2 repository directly. diff --git a/doc/plugins/generators/packages.txt b/doc/plugins/generators/packages.txt new file mode 100644 index 000000000..837a6b14a --- /dev/null +++ b/doc/plugins/generators/packages.txt @@ -0,0 +1,199 @@ +.. -*- mode: rst -*- + +======== +Packages +======== + +This page documents the Packages plugin. Packages is an alternative to [wiki:Plugins/Pkgmgr Pkgmgr] for specifying package entries for clients. Where Pkgmgr explicitly specifies package entry information, Packages delegates control of package version information to the underlying package manager, installing the latest version available through those channels. + +"Magic Groups" +============== + +Packages is the only plugin that uses "magic groups". Most plugins operate based on client group memberships, without any concern for the particular names chosen for groups by the user. The Packages plugin is the sole exception to this rule. Packages needs to "know" two different sorts of facts about clients. The first is the basic OS/distro of the client, enabling classes of sources. The second is the architecture of the client, enabling sources for a given architecture. In addition to these magic groups, each source may also specify a non-magic group to limit the source's applicability to group member clients. + ++-----------+----------+--------------+ +| Source | OS Group | Architecture | ++===========+==========+==============+ +| APTSource | debian | i386 | ++-----------+----------+--------------+ +| APTSource | ubuntu | amd64 | ++-----------+----------+--------------+ +| APTSource | nexenta | | ++-----------+----------+--------------+ +| YUMSource | redhat | i386 | ++-----------+----------+--------------+ +| YUMSource | centos | x86_64 | ++-----------+----------+--------------+ + +Limiting sources to groups +========================== + +Each source can also specify explicit group memberships. In the following example, the ubuntu-hardy group is also required. Finally, clients must be a member of the appropriate architecture group, in this case, either i386 or amd64. In total, in order for this source to be associated with a client is for the client to be in one of the sentinel groups (debian, ubuntu, or nexenta), the explicit group ubuntu-hardy, and any of the architecture groups (i386 or amd64). + +Memberships in architecture groups is needed so that Packages can map software sources to clients. There is no other way to handle this than to impose + +When multiple sources are specified, clients are associated with each source to which they apply (based on group memberships, as described above). Packages and dependencies are resolved from all applicable sources. + +Setup +===== +Three basic steps are required for Packages to work properly. + 1. Create Packages/config.xml. This file should look approximately like the example below, and describes both which software repositories should be used, and which clients are eligible to use each one. + 2. Ensure that clients are members of the proper groups. Each client should be a member of one of the sentinel groups listed above (debian/ubuntu/redhat/centos/nexenta), all of the groups listed in the source (like ubuntu-intrepid or centos-5.2 in the following examples), and one of the architecture groups listed in the source configuration (i386, amd64 or x86_64 in the following examples). '''Failure to do this will result in the source either not applying to the client, or only architecture independent packages being made available to the client.''' + 3. Add Package entries to bundles. + 4. Sit back and relax, as dependencies are resolved, and automatically added to client configurations. + +Prerequisite Resolution +======================= + +Packages provides a prerequisite resolution mechanism which has no analogue in Pkgmgr. During configuration generation, all structures are processed. After this phase, but before entry binding, a list of packages and the client metadata instance is passed into Packages' resolver. This process determines a superset of packages that will fully satisfy dependencies of all package entries included in structures, and reports any prerequisites that cannot be satisfied. This facility should largely remove the need to use the [wiki:Plugins/Base Base] plugin. + +Example usage +============= + +Create a config.xml file in the Packages directory that looks something like this: + +.. code-block:: xml + + <Sources> + <APTSource> + <Group>ubuntu-intrepid</Group> + <URL>http://us.archive.ubuntu.com/ubuntu</URL> + <Version>intrepid</Version> + <Component>main</Component> + <Component>universe</Component> + <Arch>i386</Arch> + <Arch>amd64</Arch> + </APTSource> + </Sources> + +Yum sources can be similarly specified: + +.. code-block:: xml + + <Sources> + <YUMSource> + <Group>centos-5.2</Group> + <URL>http://mirror.centos.org/centos/</URL> + <Version>5.2</Version> + <Component>os</Component> + <Component>updates</Component> + <Component>extras</Component> + <Arch>i386</Arch> + <Arch>x86_64</Arch> + </YUMSource> + </Sources> + +Configuration Updates +===================== + +Packages will reload its configuration upon an explicit command via bcfg2-admin.:: + + [0:3711] bcfg2-admin xcmd Packages.Refresh + True + +During this command (which will take some time depending on the quantity and size of the sources listed in the configuration file), the server will report information like:: + + Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-i386/Packages.gz + Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-amd64/Packages.gz + Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-i386/Packages.gz + Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-amd64/Packages.gz + ... + Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/filelists.xml.gz + Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/primary.xml.gz + +Once line per file download needed. Packages/config.xml will be reloaded at this time, so any source specification changes (new or modified sources in this file) will be reflected by the server at this point. + +Availability +============ + +Packages was added in 1.0pre1, with APT support. YUM support was added for 1.0pre2. Support for other package managers (Portage, Zypper, IPS, etc) remain to be added. + +Validation +========== + +A schema for Packages/config.xml is included; config.xml can be validated using bcfg2-repo-validate. Note that the schema requires that elements be specified in the above order. + +Limitations +=========== + +Packages does not do traditional caching as other plugins do. Changes to the Packages config file require a server restart for the time being. + +Package Verification +==================== + +In order to do disable per-package verification Pkgmgr style, you will need to use [wiki:BoundEntry BoundEntries] like below + +.. code-block:: xml + + <BoundPackage name="mem-agent" priority="1" version="auto" type="yum" verify="false"/> + +Debugging unexpected behavior +============================= + +Using bcfg2-info +---------------- + +The dependency resolver used in Packages can be run in debug mode:: + + + $ bcfg2-info + ... + Handled 20 events in 0.004s + > debug + dropping to python interpreter; press ^D to resume + ... + (debug_shell) + >>> m = self.build_metadata('ubik3') + >>> self.plugins['Packages'].complete(m, ['ssh'], debug=True) + Package ssh: adding new deps ['openssh-client', 'openssh-server'] + Package openssh-server: adding new deps ['libc6', 'libcomerr2', 'libkrb53', 'libpam0g', 'libselinux1', 'libssl0.9.8 + ', 'libwrap0', 'zlib1g', 'debconf', 'libpam-runtime', 'libpam-modules', 'adduser', 'dpkg', 'lsb-base'] + Package debconf: adding new deps ['debconf-i18n'] + Package libpam-modules: adding new deps ['libdb4.7'] + Package openssh-client: adding new deps ['libedit2', 'libncurses5', 'passwd'] + Package lsb-base: adding new deps ['sed', 'ncurses-bin'] + Package adduser: adding new deps ['perl-base'] + Package debconf-i18n: adding new deps ['liblocale-gettext-perl', 'libtext-iconv-perl', 'libtext-wrapi18n-perl', 'libtext-charwidth-perl'] + Package passwd: adding new deps ['debianutils'] + Package libtext-charwidth-perl: adding new deps ['perlapi-5.10.0'] + VPackage perlapi-5.10.0: got provides ['perl-base'] + Package libkrb53: adding new deps ['libkeyutils1'] + Package libtext-iconv-perl: adding new deps ['perlapi-5.10.0'] + Package libc6: adding new deps ['libgcc1', 'findutils'] + Package libgcc1: adding new deps ['gcc-4.3-base'] + (set(['debconf', 'libgcc1', 'lsb-base', 'libtext-wrapi18n-perl', 'libtext-iconv-perl', 'sed', 'passwd', 'findutils', 'libpam0g', 'openssh-client', 'debconf-i18n', 'libselinux1', 'zlib1g', 'adduser', 'libwrap0', 'ncurses-bin', 'libssl0.9.8', 'liblocale-gettext-perl', 'libkeyutils1', 'libpam-runtime', 'libpam-modules', 'openssh-server', 'libkrb53', 'ssh', 'libncurses5', 'libc6', 'libedit2', 'libcomerr2', 'dpkg', 'perl-base', 'libdb4.7', 'libtext-charwidth-perl', 'gcc-4.3-base', 'debianutils']), set([]), 'deb') + +This will show why the resolver is acting as it is. Replace "ubik3" and ['ssh'] with a client name and list of packages, respectively. Also, a more polished interface to this functionality is coming as well. + +Each line starting with Package: <name> describes a set of new prerequisites pulled in by a package. Lines starting with VPackage <vname> describe provides entries and their mappings to required names. The last line describes the overall results of the resolver, with three fields: a list of packages that should be installed, a list of unresolved requirements, and a type for these packages. + +Using bcfg2-server +------------------ + +Once the server is started, enable debugging via bcfg2-admin:: + + $ bcfg2-admin xcmd Packages.toggle_debug + +TODO list +========= + + * Zypper support + * Portage support + * Explicit version pinning (a la Pkgmgr) + +Developing for Packages +======================= + +In order to support a given client package tool driver, that driver must support use of the auto value for the version attribute in Package entries. In this case, the tool driver views the current state of available packages, and uses the underlying package manager's choice of correct package version in lieu of an explicit, centrally-specified, version. This support enables Packages to provide a list of Package entries with version='auto'. Currently, the APT and YUMng drivers support this feature. Note that package management systems without any network support cannot operate in this fashion, so RPMng and SYSV will never be able to use Packages. Emerge, Zypper, IPS, and Blastwave all have the needed features to be supported by Packages, but support has not yet been written. + +Packages fills two major functions in configuration generation. The first is to provide entry level binding support for Package entries included in client configurations. This function is quite easy to implement; Packages determines (based on client group membership) if the package is available for the client system, and which type it has. Because version='auto' is used, no version determination needs to be done. + +The second major function is more complex. Packages ensures that client configurations include all package-level prerequisites for package entries explicitly included in the configuration. In order to support this, Packages needs to directly process network data for package management systems (the network sources for apt or yum, for examples), process these files, and build data structures describing prerequisites and the providers of those functions/paths. To simplify implementations of this, there is a generic base class (Bcfg2.Server.Plugins.Packages.Source) that provides a framework for fetching network data via HTTP, processing those sources (with subclass defined methods for processing the specific format provided by the tool), a generic dependency resolution method, and a caching mechanism that greatly speeds up server/bcfg2-info startup. + +Each source type must define: + * a get_urls attribute (and associated urls property) that describes the URLS where to get data from. + * a read_files method that reads and processes the downloaded files + +Sources may define a get_provides method, if provides are complex. For example, provides in rpm can be either rpm names or file paths, so multiple data sources need to be multiplexed. + +The APT source in src/lib/Server/Plugins/Packages.py provides a relatively simple implementation of a source. diff --git a/doc/plugins/generators/pkgmgr.txt b/doc/plugins/generators/pkgmgr.txt new file mode 100644 index 000000000..b6940f9b1 --- /dev/null +++ b/doc/plugins/generators/pkgmgr.txt @@ -0,0 +1,237 @@ +.. -*- mode: rst -*- + +====== +Pkgmgr +====== + +.. note:: See [wiki:ClientTools/RPMng#PackageTagNewStyleandAttributes RPMng#PackageTagNewStyleandAttributes].''' The way of showing the architecture of the RPM has changed. The new way is "arch". The old way is "multiarch". '''This document needs to be updated and include version of Bcfg2 where change took place.''' + +The Pkgmgr plugin resolves the Abstract Configuration Entity "Package" to a package specification that the client can use to detect, verify and install the specified package. + +For a package specification to be included in the Literal configuration the name attribute from an Abstract Package Tag (from Base or Bundler) must match the name attribute of a Package tag in Pkgmgr, along with the appropriate group associations of course. + +Each file in the Pkgmgr directory has a priority. This allows the same package to be served by multiple files. The priorities can be used to break ties in the case that multiple files serve data for the same package. + +Usage of Groups in Pkgmgr +========================= + +Groups are used by the Pkgmgr plugin, along with host metadata, for selecting the package entries to include in the clients literal configuration. They can be thought of as:: + + if client is a member of group1 then + assign to literal config + +Nested groups are conjunctive (logical and).:: + + if client is a member of group1 and group2 then + assign to literal config + +Group membership may be negated. + +Tag Attributes in Pkgmgr +======================== + +PackageList Tag +--------------- + +The PackageList Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| priority || Sets the priority for packages in the package list.The high value wins. || Integer || +|| type || Package type that applies to all packages in the list. This value is inherited by all packages without an explicit type attribute. || (deb|rpm|blast|encap|sysv|portage|yum) || +|| uri || URI to prepend to filenames to fetch packages in this list. || String || +|| multiarch || Comma separated list of architectures that apply to all packages in this list. Inherited by all package entries in the file that do not have this attribute explicitly. || String || +|| srcs || To be used with multiarch support. Inherited by all Package entries without this attribute. || String || + +Pkgmgr Group Tag +---------------- + +The Pkgmgr Group Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Group Name || String || +|| negate || Negate group membership (is not a member of) || (True|False) || + +Package Tag +----------- + +The Package Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Package Name || String || +|| version || Package Version, version='auto' to install the latest version in the client's cache, or version='any' to verify that any version of the package is installed on the client || String || +|| file || Package file name. Several other attributes (name, version) can be automatically defined based on regular expressions defined in the Pkgmgr plugin. || String || +|| simplefile || Package file name. No name parsing is performed, so no extra fields get set || String || +|| verify || verify='false' - do not do package verification || String || +|| multiarch || Comma separated list of the architectures of this package that should be installed. (Temporary Work around) || String || +|| srcs || File name creation rules for multiarch packages. (Temporary Work around) || String || +|| type || Package type. (rpm, yum, apt,sysv,blast) || String || + +Client Tag +---------- + +The Client Tag is used in a PackageList for selecting the package entries to include in the clients literal configuration. Its function is similar to the Group tag in this context. It can be thought of as:: + + if client is name then + assign to literal config + +The Client Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Client Name || String || +|| negate || Negate client selection (if not client name) || (True|False) || + +Pkgmgr Directory +================ + +The Pkgmgr/ directory keeps the XML files that define what packages are available for a host or image and where to find those packages. All the files in the directory are processed. + +The names of the XML files have no special meaning to Bcfg2; they are simply named so it's easy for the administrator to know what the contents hold. All Packages could be kept in a single file if so desired. Bcfg2 simply uses the Groups in the files and priorities to determine how to assign Packages to a host's literal configuration. + +Listed detailed below is one possible structure for the Pkgmgr directory. + +The files are structured to contain particular portions of distribution repositories. + +The files in the directory are:: + + $ ls Pkgmgr/ + centos-4-noarch-updates.xml + centos-4-x86_64-updates.xml + centos-4-x86_64.xml + backup.example.com.xml + fedora-core-4-noarch-updates.xml + fedora-core-4-x86-updates.xml + fedora-core-4-x86.xml + rhel-as-4-noarch-updates.xml + rhel-as-4-x86-updates.xml + rhel-as-4-x86.xml + rhel-es-4-noarch-updates.xml + rhel-es-4-x86-updates.xml + rhel-es-4-x86.xml + rhel-ws-4-noarch-udpates.xml + rhel-ws-4-x86_64-updates.xml + rhel-ws-4-x86_64.xml + rhel-ws-4-x86-updates.xml + rhel-ws-4-x86.xml + +As can be seen the file names have been selected to indicate what the contents are and have been split by Vendor, product and repository area. + +A partial listing of the centos-4-x86_64.xml is below + +.. code-block:: xml + + $ cat centos-4-x86_64.xml + <PackageList uri='http://www.example.com/yam/Centos44-x86_64/RPMS.os/' type='yum' priority='0'> + <Group name='Centos4.4-Standard'> + <Group name='x86_64'> + <Package name='audit-libs' version='1.0.13-1.EL4' type='yum'/> + <Package name='audit' version='1.0.13-1.EL4' type='yum'/> + <Package name='basesystem' version='8.0-2' type='yum'/> + <Package name='bash' version='3.0-18.1' type='yum'/> + <Package name='bcfg2' version='0.9.1-0.1' type='yum'/> + <Package name='beecrypt' version='3.1.0-3' type='yum'/> + ... + <Package name='VMwareTools' version='6530-29996' type='yum'/> + <Package name='yum' version='2.4.2-1' type='yum'/> + <Package name='zlib' version='1.2.1.2-1.2' type='yum'/> + </Group> + </Group> + </PackageList> + +.. code-block:: xml + + $ cat centos-4-x86_64-updates.xml + <PackageList uri='http://www.example.com/yam/Centos44-x86_64/RPMS.updates/' type='yum' priority='5'> + <Group name='Centos4.4-Standard'> + <Group name='x86_64'> + <Package name='audit-libs' version='1.0.14-1.EL4' type='yum'/> + <Package name='audit' version='1.0.14-1.EL4' type='yum'/> + <Package name='basesystem' version='8.0-4' type='yum'/> + <Package name='bash' version='3.0-19.3' type='yum'/> + <Package name='bcfg2' version='0.9.2-0.1' type='yum'/> + <Package name='beecrypt' version='3.1.0-6' type='yum'/> + ... + <Package name='VMwareTools' version='6530-29996' type='yum'/> + <Package name='yum' version='2.4.3-1' type='yum'/> + <Package name='zlib' version='1.2.1.2-1.2' type='yum'/> + </Group> + </Group> + </PackageList> + +Here it can be seen that the data is encapsulated in a !PackageList Tag which describes the URI of the files described, the type of package, and the priority of the files in this list. + +The priority is used to decide which specific file to use when there are multiple files that could be used for a particular host. The highest priority file is the one that is used. + +Using this system, it is possible to have a file that contains all the Packages from the original installation, centos-4-x86_64.xml in this case, and then create a new file that contains updates that are made available afterwards, centos-4-x86_64-updates.xml and centos-4-noarch-updates.xml in this case. The priority of the update PackageLists just needs to be higher so that they will be selected instead of the original installation Packages. + +The backup.example.com.xml contains a packalist for a specific host which is qualified by the Client tag. Its Packages have a higher priority than the update Packages. This is because this particular host requires special Packages that are older than the ones available in the updates. + +.. code-block:: xml + + <PackageList uri='http://www.example.com/yam/Centos44-x86_64/RPMS.os/' type='yum' priority='1000'> + <Client name='server86.example.com'> + <Package name='audit-libs' version='1.0.13-1.EL4' type='yum'/> + <Package name='audit' version='1.0.13-1.EL4' type='yum'/> + <Package name='basesystem' version='8.0-2' type='yum'/> + <Package name='bash' version='3.0-18.1' type='yum'/> + <Package name='bcfg2' version='0.9.1-0.1' type='yum'/> + <Package name='beecrypt' version='3.1.0-3' type='yum'/> + ... + <Package name='VMwareTools' version='6530-29996' type='yum'/> + <Package name='yum' version='2.4.2-1' type='yum'/> + <Package name='zlib' version='1.2.1.2-1.2' type='yum'/> + </Client> + </PackageList> + +Simplifying Multi-Architecture Environments with [wiki:altsrc Altsrc] +===================================================================== + +This feature is available in 0.9.5pre3 or newer versions of the bcfg2 server. + +Frequently multi-architecture environments (typically x86_64) will run into problems needing to specify different architectures on different groups for clients. For example, desktop machines may install 32-bit compatibility packages in addition to 64-bit ones, while servers may install only 64-bit packages. Specifying this in the Pkgmgr was onerous, because different package targets (64bit, 32+64, etc) needed to be specified on a package by group basis. Two features have been implemented that should ease this situation considerably. +* the [wiki:altsrc] feature adds the ability to add a "bind as" directive to entries. For example, the following entry, in a bundle: + +.. code-block:: xml + + <Package name='foo' altsrc='bar'/> + + would bind as if it were named bar, while the entry would still appear named "foo" in the client configuration specification. + +* Pkgmgr now builds virtual package targets for any package with Instance client elements. This means that if a client attempts to bind: + +.. code-block:: xml + + <Package name='libfoo:x86_64,i686'/> + It will only include the instances listed in the package. + By using these features together, a bundle can include: + +.. code-block:: xml + + <Package name='libfoo' altsrc='libfoo:i386,i686'/> + + This in conjunction with a Pkgmgr entry that looks like: + +.. code-block:: xml + + <Package name='libfoo'> + <Instance arch='i386' version='1.0.4-12'/> + <Instance arch='i586' version='1.0.4-12'/> + <Instance arch='i686' version='1.0.4-12'/> + <Instance arch='x86_64' version='1.0.4-12'/> + </Package> + + Will result in a bound entry that looks like: + +.. code-block:: xml + + <Package name='libfoo'> + <Instance arch='i386' version='1.0.4-12'/> + <Instance arch='i686' version='1.0.4-12'/> + </Package> + +Altogether, this should move policy decisions about package architectures to bundles/base. + +Client Driver (Plugins) Specific Attributes +=========================================== + +Not all the attributes that are available in Pkgmgr are honoured by all the client drivers. The following client drivers (plugins) have driver specific attributes: +* [wiki:RPMng RPMng/YUMng] diff --git a/doc/plugins/generators/rules.txt b/doc/plugins/generators/rules.txt new file mode 100644 index 000000000..94510d91f --- /dev/null +++ b/doc/plugins/generators/rules.txt @@ -0,0 +1,199 @@ +.. -*- mode: rst -*- + +===== +Rules +===== + +The Rules plugin resolves the following Abstract Configuration Entities: + +* Service +* Directory +* SymLink +* Package +* Path +* Permissions +* Action + +to literal configuration entries suitable for the client drivers to consume. + +For an entity specification to be included in the Literal configuration the name attribute from an Abstract Entity Tag (from Base or Bundler) must match the name attribute of an Entity tag in Rules, along with the appropriate group associations of course. + +Each file in the Rules directory has a priority. This allows the same Entities to be served by multiple files. The priorities can be used to break ties in the case that multiple files serve data for the same Entity. + + +Usage of Groups in Rules +======================== + +Groups are used by the Rules plugin, along with host metadata, for selecting the Configuration Entity entries to include in the clients literal configuration. They can be thought of as:: + + if client is a member of group1 then + assign to literal config + +Nested groups are conjunctive (logical and).:: + + if client is a member of group1 and group2 then + assign to literal config + +Group membership may be negated. + +Tag Attributes in Rules +======================= + +Rules Tag +--------- + +The Rules Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| priority || Sets the priority for Rules in this Rules list.The higher value wins. || String || + +Rules Group Tag +--------------- + +The Rules Group Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Group Name || String || +|| negate || Negate group membership (is not a member of) || (True|False) || + +Package Tag +----------- + +The Package Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Package Name || String || +|| version || Package Version orversion='noverify' to not do version checking in the Yum driver only (temporary work a round). || String || +|| file || Package file name. Several other attributes (name, version) can be automatically defined based on regular expressions defined in the Pkgmgr plugin. || String || +|| simplefile || Package file name. No name parsing is performed, so no extra fields get set || String || +|| verify || verify='false' - do not do package verification || String || +|| reloc || RPM relocation path. || String || +|| multiarch || Comma separated list of the architectures of this package that should be installed. || String || +|| srcs || Filename creation rules for multiarch packages. || String || +|| type || Package type. (rpm, yum, apt,sysv,blast) || String || + +Permissions Tag +--------------- + +The Permissions tag is + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Name of the file. || String || +|| perms || Permissions of the file. || || +|| owner || Owner of the file. || || +|| group || Group of the file. || || + +Action Tag +---------- + +See [wiki:Plugins/Actions Actions] + +Service Tag +----------- + +The Service tag is + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Service Name || String || +|| status || Should the service be on or off. || (on|off) || +|| restart || Service command for restart (default:restart) || String || +|| type || Driver to use on the client to manage this service. || (chkconfig|deb|rc-update|smf) || +|| sequence || Order for service startup (debian services only) || integer || + +Directory Tag +------------- + +The Directory tag is + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Directory Name || String || +|| perms || Permissions of the directory. || String || +|| owner || Owner of the directory || String || +|| group || Group Owner of the directory || String || +|| prune || prune unspecified entries from the Directory || true|false || + +SymLink Tag +----------- + +The SymLink tag is + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Name of the symlink. || String || +|| to || File to link to || String || + +Client Tag +---------- + +The Client Tag is used in Rules for selecting the package entries to include in the clients literal configuration. Its function is similar to the Group tag in this context. It can be thought of as:: + + if client is name then + assign to literal config + +The Client Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Client Name || String || +|| negate || Negate client selection (if not client name) || (True|False) || + +Path Tag +-------- + +The Path tag is + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Name of the file || String || +|| type || Type of file || nonexistent || + + +Rules Directory +=============== + +The Rules/ directory keeps the XML files that define what rules are available for a host. All the files in the directory are processed. + +The names of the XML files have no special meaning to Bcfg2; they are simply named so it's easy for the administrator to know what the contents hold. All Rules could be kept in a single file if so desired. Bcfg2 simply uses the Groups in the files and priorities to determine how to assign Rules to a host's literal configuration. + +.. code-block:: xml + + <Rules priority="0"> + <Directory group="root" name="/autonfs" owner="root" perms="0755"/> + <Directory group="utmp" name="/var/run/screen" owner="root" perms="0775"/> + <Directory group="root" name="/autonfs/stage" owner="root" perms="0755"/> + <Directory group="root" name="/exports" owner="root" perms="0755"/> + <Directory name="/etc/condor" owner="root" group="root" perms="0755"/> + <Directory name="/logs" group="wwwtrans" owner="root" perms="0775"/> + <Directory name="/mnt" group="root" owner="root" perms="0755"/> + <Directory name="/my" owner="root" group="root" perms="0755"/> + <Directory name="/my/bin" owner="root" group="root" perms="0755"/> + <Directory name="/nfs" owner="root" group="root" perms="0755"/> + <Directory name="/sandbox" perms="0777" owner="root" group="root"/> + <Directory name="/software" group="root" owner="root" perms="0755"/> + <Permissions perms="0555" group="audio" owner="root" name="/dev/dsp"/> + <Permissions perms="0555" group="audio" owner="root" name="/dev/mixer"/> + <SymLink name="/bin/whatami" to="/mcs/adm/bin/whatami"/> + <SymLink name="/chibahomes" to="/nfs/chiba-homefarm"/> + <SymLink name="/home" to="/nfs/mcs-homefarm"/> + <SymLink name="/homes" to="/home"/> + <SymLink name="/mcs" to="/nfs/mcs"/> + <SymLink name="/my/bin/bash" to="/bin/bash"/> + <SymLink name="/my/bin/tcsh" to="/bin/tcsh"/> + <SymLink name="/my/bin/zsh" to="/bin/zsh"/> + <SymLink name="/software/common" to="/nfs/software-common"/> + <SymLink name="/software/linux" to="/nfs/software-linux"/> + <SymLink name="/software/linux-debian_sarge" to="/nfs/linux-debian_sarge"/> + <SymLink name="/usr/bin/passwd" to="/usr/bin/yppasswd"/> + <SymLink name="/usr/bin/yppasswd" to="/mcs/bin/passwd"/> + <SymLink name="/usr/lib/libgd.so.1.8" to="/usr/lib/libgd.so.1.8.4"/> + <SymLink name="/usr/lib/libtermcap.so.2" to="/usr/lib/libtermcap.so"/> + <SymLink name="/usr/local/bin/perl" to="/usr/bin/perl"/> + <SymLink name="/usr/local/bin/perl5" to="/usr/bin/perl"/> + <SymLink name="/usr/local/bin/tcsh" to="/bin/tcsh"/> + <Service name='ntpd' status='on' type='chkconfig'/> + <Service name='haldaemon' status='on' type='chkconfig'/> + <Service name='messagebus' status='on' type='chkconfig'/> + <Service name='netfs' status='on' type='chkconfig'/> + <Service name='network' status='on' type='chkconfig'/> + <Service name='rawdevices' status='on' type='chkconfig'/> + <Service name='sshd' status='on' type='chkconfig'/> + <Service name='syslog' status='on' type='chkconfig'/> + <Service name='vmware-tools' status='on' type='chkconfig'/> + </Rules> diff --git a/doc/plugins/generators/sshbase.txt b/doc/plugins/generators/sshbase.txt new file mode 100644 index 000000000..65fe1cca7 --- /dev/null +++ b/doc/plugins/generators/sshbase.txt @@ -0,0 +1,38 @@ +.. -*- mode: rst -*- + +======= +SSHbase +======= + +SSHbase is a purpose build bcfg2 plugin for managing ssh host keys. It is responsible for making ssh keys persist beyond a client rebuild and building a proper ssh_known_hosts file, including a correct localhost record for the current system. + +It has two functions: + +* Generating new ssh keys -- When a client requests a dsa, rsa, or v1 key, and there is no existing key in the repository, one is generated. +* Maintaining the ssh_known_hosts file -- all current known public keys (and extra public key stores) are integrated into a single ssh_known_hosts file, and a localhost record for the current client is added. The ssh_known_hosts file data is updated whenever any keys change, are added, or deleted. + +Interacting with SSHbase +======================== + +* Pre-seeding with existing keys -- Currently existing keys will be overwritten by new, sshbase-managed ones by default. Pre-existing keys can be added to the repository by putting them in <repo>/SSHbase/<key filename>.H_<hostname> +* Pre-seeding can also be performed using bcfg2-admin pull ConfigFile /name/of/ssh/key +* Revoking existing keys -- deleting <repo>/SSHbase/\*.H_<hostname> will remove keys for an existing client. + +Aliases +======= + +As of 1.0pre4, SSHbase has support for Aliases listed in clients.xml. The address for the entries are specified either through DNS (e.g. a CNAME), or via the address attribute to the Alias. + +Getting started +=============== + +#. Add SSHbase to the generators line (plugins line in 1.0 or greater) in /etc/bcfg2.conf and restart the server -- This enables the SSHbase plugin in the bcfg2 server. +#. Add ConfigFile entries for /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_host_dsa_key, etc to a bundle or base. +#. Enjoy. + +At this point, SSHbase will generate new keys for any client without a recorded key in the repository, and will generate an ssh_known_hosts file appropriately. + +Blog post +========= + +[http://www.ducea.com/2008/08/24/using-the-bcfg2-sshbase-plugin/] diff --git a/doc/plugins/generators/tcheetah.txt b/doc/plugins/generators/tcheetah.txt new file mode 100644 index 000000000..5dcc466c0 --- /dev/null +++ b/doc/plugins/generators/tcheetah.txt @@ -0,0 +1,149 @@ +.. -*- mode: rst -*- + +======== +TCheetah +======== + +This document reflects the [source:trunk/bcfg2/src/lib/Server/Plugins/TCheetah.py TCheetah plugin] in bcfg2 0.8.4 and later. + +The TCheetah plugin allows you to use the [http://www.cheetahtemplate.org/ cheetah templating system] to create files, instead of the various diff-based methods offered by the Cfg plugin. It also allows you to include the results of probes executed on the client in the created files. + +To begin, you will need to download and install the Cheetah templating engine from [http://www.cheetahtemplate.org/]. Once it is installed, you can enable it by adding `TCheetah` to the `plugins` line in `/etc/bcfg2.conf` on your Bcfg server. +For example:: + + generators = SSHbase,Cfg,Pkgmgr,Svcmgr,Rules,TCheetah + +The TCheetah plugin makes use of a Cfg-like directory structure located in in a `TCheetah` subdirectory of your repository, usually `/var/lib/bcfg2/TCheetah`. Each file has a directory containing two files, `template` and `info`. The template is a standard Cheetah template with two additions: + +* `self.metadata` is the client's metadata +* `self.properties` is an xml document of unstructured data + +The `info` file is formatted like `:info` files from Cfg. + +Mostly, people will want to use client metadata. + +self.metadata variables +======================= + +The following variables are available for self.metadata: + +* hostname +* bundles +* groups +* toolset +* categories +* probes +* uuid +* password + +self.metadata is an instance of the class ClientMetadata of file [http://trac.mcs.anl.gov/projects/bcfg2/browser/trunk/bcfg2/src/lib/Server/Plugins/Metadata.py Bcfg2/Server/Plugins/Metadata.py]. + +self.properties +=============== + +properties is a python [http://codespeak.net/lxml/ ElementTree] object, loaded from the data in /var/lib/bcfg2/etc/properties.xml. That file should have a Properties node at its root. + +Example properties.xml: + +.. code-block:: xml + + <Properties> + <host> + <www.example.com> + <rootdev>/dev/sda</rootdev> + </www.example.com> + </host> + </Properties> + +You may use any of the ElementTree methods to access data in your template. Several examples follow, each producing an identical result on the host 'www.example.com':: + + $self.properties.find('host').find('www.example.com').find('rootdev').text + $self.properties.find('host').find($self.metadata.hostname).find('rootdev').text + ${self.properties.xpath('host/www.example.com/rootdev')[0].text} + ${self.properties.xpath('host/' + self.metadata.hostname + '/rootdev')[0].text} + #set $path = 'host/' + $self.metadata.hostname + '/rootdev' + ${self.properties.xpath($path)[0].text} + ${self.properties.xpath(path)[0].text} + +Simple Example +============== + +bcfg2/TCheetah/foo/template +--------------------------- + +.. code-block:: none + + > buildfile /foo <clientname> + Hostname is $self.metadata.hostname + Groups: + #for $group in $self.metadata.groups: + * $group + #end for + Categories: + #for $category in $self.metadata.categories: + * $category -- $self.metadata.categories[$category] + #end for + + Probes: + #for $probe in $self.metadata.probes: + * $probe -- $self.metadata.probes[$probe] + #end for + +bcfg2/TCheetah/foo/info +----------------------- + +.. code-block:: none + + perms: 624 + +Output +------ + +The following output can be generated with bcfg2-info. Note that probe information is not persistent, hence, it only works when clients directly query the server. For this reason, bcfg2-info output doesn't reflect current client probe state. + +.. code-block:: xml + + <Path type="file" name="/foo" owner="root" perms="0624" group="root"> + Hostname is topaz.mcs.anl.gov + Groups: + * desktop + * mcs-base + * ypbound + * workstation + * xserver + * debian-sarge + * debian + * a + Categories: + * test -- a + + Probes: + </Path> + +Example: Replace the crontab plugin +=================================== + +In many cases you can use the TCheetah plugin to avoid writing custom plugins in Python. This example replaces the [source:tags/bcfg2_0_8_4/bcfg2/src/lib/Server/Plugins/Crontab.py crontab plugin] (Bcfg2.Server.Plugins.Crontab). This plugin randomizes the time of cron.daily execution with a stable result. Cron.daily is run at a consistent, randomized time between midnight and 7am.:: + + #import random + #silent random.seed($self.metadata.hostname) + + # /etc/crontab: system-wide crontab + # Unlike any other crontab you don't have to run the `crontab` + # command to install the new version when you edit this file. + # This file also has a username field, that none of the other crontabs do. + + SHELL=/bin/sh + PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin://bin + + # m h dom mon dow user command + 17 * * * * root run-parts --report /etc/cron.hourly + $random.randrange(0,59) $random.randrange(0,6) * * * root test -x /usr/sbin/anacron || run-parts --report /etc/cron.daily + 47 6 * * 7 root test -x /usr/sbin/anacron || run-parts --report /etc/cron.weekly + 52 6 1 * * root test -x /usr/sbin/anacron || run-parts --report /etc/cron.monthly. + +.. note:: Comments and Cheetah + As Cheetah processes your templates it will consider hash "#" style comments to be actual comments in the template and will strip them from the final config file. If you would like to preserve the comment in the final config file you need to escape the hash character '\#' which will tell Cheetah (and Python) that you do in fact want the comment to appear in the final config file.:: + + # This is a comment in my template which will be stripped when it's processed through Cheetah + \# This comment will appear in the generated config file. diff --git a/doc/plugins/generators/tgenshi.txt b/doc/plugins/generators/tgenshi.txt new file mode 100644 index 000000000..5794bac25 --- /dev/null +++ b/doc/plugins/generators/tgenshi.txt @@ -0,0 +1,73 @@ +.. -*- mode: rst -*- + +======= +TGenshi +======= + +This page documents the TGenshi plugin. This plugin works with version 0.4 and newer of the genshi library. + +The TGenshi plugin allows you to use the [http://genshi.edgewall.org Genshi] templating system to create files, instead of the various diff-based methods offered by the Cfg plugin. It also allows you to include the results of probes executed on the client in the created files. + +To begin, you will need to download and install the Genshi templating engine. + +To install on CentOS or RHEL 5, run:: + + sudo yum install python-genshi + +Once it is installed, you can enable it by adding TGenshi to the generators line in /etc/bcfg2.conf on your Bcfg server. For example:: + + generators = SSHbase,Cfg,Pkgmgr,Svcmgr,Rules,TGenshi + +The TGenshi plugin makes use of a Cfg-like directory structure located in in a TGenshi subdirectory of your repository, usually `/var/lib/bcfg2/TGenshi`. Each file has a directory containing two file types, template and info. Templates are named according to the genshi format used; template.txt uses the genshi text format, and template.xml uses the XML format. + +If used with Genshi 0.5 or later the plugin also supports the [http://genshi.edgewall.org/wiki/Documentation/0.5.x/text-templates.html new style] text template format for files named template.newtxt. One of the advantages of the new format is that it does not use # as a command delimiter, making it easier to utilize for configuration files that use # as a comment character. + +Only one template format may be used per file served. Info files are identical to those used in Cfg, and info.xml files are supported. + +Inside of templates +=================== + +* metadata is the client's metadata +* properties.properties is an xml document of unstructured data + +See the genshi [http://genshi.edgewall.org/wiki/Documentation documentation] for examples of Genshi syntax. + +Examples +======== + +See [wiki:Plugins/TGenshi/examples] for some TGenshi template examples, including the great starting point [wiki:Plugins/TGenshi/examples/motd /etc/motd]. They will use the new style of Genshi syntax, unless noted otherwise. + +Examples: Old Genshi Syntax +--------------------------- + +Genshi's web pages recommend against using this syntax, as it may disappear from future releases. + + +Group Negation +-------------- + +Templates are also useful for cases where more sophisticated boolean operations than those supported by Cfg are needed. For example, the template:: + + #if "ypbound" in metadata.groups and "workstation" in metadata.groups + client is ypbound workstation + #end + #if "ubuntu" not in metadata.groups and "desktop" in metadata.groups + client is a desktop, but not an ubuntu desktop + #end + +Produces: + +.. code-block:: xml + + <Path type="file" name="/bar.conf" owner="root" perms="0644" group="root">client is ypbound workstation + client is a desktop, but not an ubuntu desktop + </Path> + +This flexibility provides the ability to build much more compact and succinct definitions of configuration contents than Cfg can. + +FAQs +==== + +'''Question:''' How do I escape the $ (dollar sign) in a TGenshi text template? For example, if I want to include SVN (subversion) keywords like $Id$ or $HeadURL$ in TGenshi-generated files, or am templating a bourne shell (sh/bash) script or Makefile (make). + +'''Answer:''' Use $$ (double dollar sign) to output a literal $ (dollarsign) in a TGenshi text template. So instead of $Id$, you'd use $$Id$$. See also Genshi tickets [http://genshi.edgewall.org/ticket/282 #282: Document $$ escape convention] and [http://genshi.edgewall.org/ticket/283 #283: Allow for redefinition of template syntax per-file]. diff --git a/doc/plugins/bb.txt b/doc/plugins/grouping/bb.txt index 806f180e8..806f180e8 100644 --- a/doc/plugins/bb.txt +++ b/doc/plugins/grouping/bb.txt diff --git a/doc/plugins/metadata.txt b/doc/plugins/grouping/metadata.txt index 2c169318f..2c169318f 100644 --- a/doc/plugins/metadata.txt +++ b/doc/plugins/grouping/metadata.txt diff --git a/doc/plugins/hostbase.txt b/doc/plugins/hostbase.txt deleted file mode 100644 index 38580598a..000000000 --- a/doc/plugins/hostbase.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -======== -Hostbase -======== diff --git a/doc/plugins/index.txt b/doc/plugins/index.txt index 0d46fc102..a478f6a8f 100644 --- a/doc/plugins/index.txt +++ b/doc/plugins/index.txt @@ -9,7 +9,7 @@ Plugins are the source of all logic used in building a config. They can perform #. Generating configuration inventory lists for clients #. Generating configuration entry contents for clients #. Probing client-side state (like hardware inventory, etc) -- the generic client probing mechanism is described at :doc:`probes`. -#. Automating administrative tasks (e.g. :doc:`sshbase` which automates ssh key management) +#. Automating administrative tasks (e.g. :doc:`generators/sshbase` which automates ssh key management) #. Generating client per-entry installation decision-lists Enabling Plugins @@ -27,53 +27,55 @@ The `Bcfg2 repository`_ has the default plugin list currently distributed with B Metadata (Grouping) ------------------- -* :doc:`bb` -* :doc:`metadata` +.. toctree:: + :maxdepth: 2 + :glob: + + grouping/* Each of these plugins has a corresponding subdirectory with the same name in the Bcfg2 repository. Abstract Configuration (Structures) ----------------------------------- -* :doc:`base` -* :doc:`bundler` +.. toctree:: + :maxdepth: 2 + :glob: + + structures/* Each of these plugins has a corresponding subdirectory with the same name in the Bcfg2 repository. Literal Configuration (Generators) ---------------------------------- -* :doc:`account` -* :doc:`actions` -* :doc:`cfg` -* :doc:`decisions` -* :doc:`deps` -* :doc:`hostbase` -* :doc:`nagiosgen` -* :doc:`packages` -* :doc:`pkgmgr` -* :doc:`rules` -* :doc:`sshbase` -* :doc:`tcheetah` -* :doc:`tgenshi` +.. toctree:: + :maxdepth: 2 + :glob: + + generators/* Each of these plugins has a corresponding subdirectory with the same name in the Bcfg2 repository. Statistics Plugins ------------------ -* :doc:`dbstats` -* :doc:`statistics` +.. toctree:: + :maxdepth: 2 + :glob: + + statistics/* DBStats can be enabled by adding it to the plugins line in /etc/bcfg2.conf. Version Plugins --------------- -* :doc:`bzr` -* :doc:`fossil` -* :doc:`git` -* :doc:`svn` +.. toctree:: + :maxdepth: 2 + :glob: + + version/* Plugin Roles (in 1.0) ===================== @@ -81,3 +83,9 @@ Plugin Roles (in 1.0) In version 1.0, plugins have been refactored into a series of roles. This are fine-grained plugin capabilities that govern how the server core interacts with plugins. More details can be found in :doc:`plugin-roles` + +.. toctree:: + :hidden: + + plugin-roles + probes diff --git a/doc/plugins/nagiosgen.txt b/doc/plugins/nagiosgen.txt deleted file mode 100644 index f157cd82b..000000000 --- a/doc/plugins/nagiosgen.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -========= -NagiosGen -========= diff --git a/doc/plugins/packages.txt b/doc/plugins/packages.txt deleted file mode 100644 index b2dbfd1ab..000000000 --- a/doc/plugins/packages.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -======== -Packages -======== diff --git a/doc/plugins/pkgmgr.txt b/doc/plugins/pkgmgr.txt deleted file mode 100644 index 63cb7de1c..000000000 --- a/doc/plugins/pkgmgr.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -====== -Pkgmgr -====== diff --git a/doc/plugins/rules.txt b/doc/plugins/rules.txt deleted file mode 100644 index 8d32df856..000000000 --- a/doc/plugins/rules.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -===== -Rules -===== diff --git a/doc/plugins/sshbase.txt b/doc/plugins/sshbase.txt deleted file mode 100644 index 28b541732..000000000 --- a/doc/plugins/sshbase.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -======= -SSHbase -======= diff --git a/doc/plugins/dbstats.txt b/doc/plugins/statistics/dbstats.txt index 4afca4afe..4afca4afe 100644 --- a/doc/plugins/dbstats.txt +++ b/doc/plugins/statistics/dbstats.txt diff --git a/doc/plugins/statistics.txt b/doc/plugins/statistics/statistics.txt index 4aae9c0f2..4aae9c0f2 100644 --- a/doc/plugins/statistics.txt +++ b/doc/plugins/statistics/statistics.txt diff --git a/doc/plugins/base.txt b/doc/plugins/structures/base.txt index ead775c10..ead775c10 100644 --- a/doc/plugins/base.txt +++ b/doc/plugins/structures/base.txt diff --git a/doc/plugins/bundler.txt b/doc/plugins/structures/bundler.txt index 1efbca878..1efbca878 100644 --- a/doc/plugins/bundler.txt +++ b/doc/plugins/structures/bundler.txt diff --git a/doc/plugins/tcheetah.txt b/doc/plugins/tcheetah.txt deleted file mode 100644 index 1b0b4907d..000000000 --- a/doc/plugins/tcheetah.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -======== -TCheetah -======== diff --git a/doc/plugins/tgenshi.txt b/doc/plugins/tgenshi.txt deleted file mode 100644 index f8acb9933..000000000 --- a/doc/plugins/tgenshi.txt +++ /dev/null @@ -1,5 +0,0 @@ -.. -*- mode: rst -*- - -======= -TGenshi -======= diff --git a/doc/plugins/bzr.txt b/doc/plugins/version/bzr.txt index 83bea745e..83bea745e 100644 --- a/doc/plugins/bzr.txt +++ b/doc/plugins/version/bzr.txt diff --git a/doc/plugins/fossil.txt b/doc/plugins/version/fossil.txt index 893ab266e..893ab266e 100644 --- a/doc/plugins/fossil.txt +++ b/doc/plugins/version/fossil.txt diff --git a/doc/plugins/git.txt b/doc/plugins/version/git.txt index 621c64127..621c64127 100644 --- a/doc/plugins/git.txt +++ b/doc/plugins/version/git.txt diff --git a/doc/plugins/svn.txt b/doc/plugins/version/svn.txt index cfe44c707..cfe44c707 100644 --- a/doc/plugins/svn.txt +++ b/doc/plugins/version/svn.txt |