diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/architecture.txt | 354 | ||||
| -rw-r--r-- | doc/contents.txt | 1 | ||||
| -rw-r--r-- | doc/index.txt | 3 | ||||
| -rw-r--r-- | doc/server/configurationentries.txt | 133 | ||||
| -rw-r--r-- | doc/server/index.txt | 4 | ||||
| -rw-r--r-- | doc/server/plugins/generators/hostbase.txt | 22 | ||||
| -rw-r--r-- | doc/server/snapshots/index.txt | 22 | ||||
| -rw-r--r-- | doc/unsorted/altsrc.txt | 102 | ||||
| -rw-r--r-- | doc/unsorted/annotated_examples.txt | 191 | ||||
| -rw-r--r-- | doc/unsorted/boundentry.txt | 15 | ||||
| -rw-r--r-- | doc/unsorted/contributors.txt | 32 | ||||
| -rw-r--r-- | doc/unsorted/doc_architecture.txt | 128 | ||||
| -rw-r--r-- | doc/unsorted/gentoo.txt | 95 | ||||
| -rw-r--r-- | doc/unsorted/groupnames.txt | 17 | ||||
| -rw-r--r-- | doc/unsorted/mailinglist.txt | 9 | ||||
| -rw-r--r-- | doc/unsorted/nisauth.txt | 19 | ||||
| -rw-r--r-- | doc/unsorted/ssl.txt | 29 | ||||
| -rw-r--r-- | doc/unsorted/writing_specification.txt | 12 |
18 files changed, 622 insertions, 566 deletions
diff --git a/doc/architecture.txt b/doc/architecture.txt new file mode 100644 index 000000000..ef2fe3830 --- /dev/null +++ b/doc/architecture.txt @@ -0,0 +1,354 @@ +.. -*- mode: rst -*- + +.. _architecture: + +=========================== +Detailed Bcfg2 Architecture +=========================== + +Bcfg2 is based on a client-server architecture. The client is responsible +for interpreting (but not processing) the configuration served by the +server. This configuration is literal, so no local process is required. +After completion of the configuration process, the client uploads a set +of statistics to the server. This section will describe the goals and +then the architecture motivated by it. + +Goals +===== + +* **Model configurations using declarative semantics.** + + Declarative semantics maximize the utility of configuration management + tools; they provide the most flexibility for the tool to determine + the right course of action in any given situation. This means that + users can focus on the task of describing the desired configuration, + while leaving the task of transitioning clients states to the tool. + +* **Configuration descriptions should be comprehensive.** + + This means that configurations served to the client should be sufficient + to reproduce all desired functionality. This assumption allows the + use of heuristics to detect extra configuration, aiding in reliable, + comprehensive configuration definitions. + +* **Provide a flexible approach to user interactions.** + + Most configuration management systems take a rigid approach to user + interactions; that is, either the client system is always correct, + or the central system is. This means that users are forced into an + overly proscribed model where the system asserts where correct data + is. Configuration data modification is frequently undertaken on both + the configuration server and clients. Hence, the existence of a single + canonical data location can easily pose a problem during normal tool + use. Bcfg2 takes a different approach. + +The default assumption is that data on the server is correct, however, +the client has the option to run in another mode where local changes are +catalogued for server-side integration. If the Bcfg2 client is run in dry +run mode, it can help to reconcile differences between current client +state and the configuration described on the server. The Bcfg2 client +also searches for extra configuration; that is, configuration that is +not specified by the configuration description. When extra configuration +is found, either configuration has been removed from the configuration +description on the server, or manual configuration has occurred on the +client. Options related to two-way verification and removal are useful +for configuration reconciliation when interactive access is used. + +* Plugins and administrative applications. +* Incremental operations. + +The Bcfg2 Client +================ + +The Bcfg2 client performs all client configuration or reconfiguration +operations. It renders a declarative configuration specification, provided +by the Bcfg2 server, into a set of configuration operations which will, +if executed, attempt to change the client's state into that described by +the configuration specification. Conceptually, the Bcfg2 client serves to +isolate the Bcfg2 server and specification from the imperative operations +required to implement configuration changes. + +This isolation allows declarative specifications to be manipulated +symbolically on the server, without needing to understand the properties +of the underlying system tools. In this way, the Bcfg2 client acts +as a sort of expert system that *knows* how to implement declarative +configuration changes. + +The operation of the Bcfg2 client is intended to be as simple as +possible. The normal configuration process consists of four main steps: + +* **Probe Execution** + + During the probe execution stage, the client connects to the server + and downloads a series of probes to execute. These probes reveal + local facts to the Bcfg2 server. For example, a probe could discover + the type of video card in a system. The Bcfg2 client returns this + data to the server, where it can influence the client configuration + generation process. + +* **Configuration Download and Inventory** + + The Bcfg2 client now downloads a configuration specification from the + Bcfg2 server. The configuration describes the complete target state + of the machine. That is, all aspects of client configuration should + be represented in this specification. For example, all software + packages and services should be represented in the configuration + specification. The client now performs a local system inventory. + This process consists of verifying each entry present in the + configuration specification. After this check is completed, heuristic + checks are executed for configuration not included in the configuration + specification. We refer to this inventory process as 2-way validation, + as first we verify that the client contains all configuration that + is included in the specification, then we check if the client has + any extra configuration that isn't present. This provides a fairly + rigorous notion of client configuration congruence. Once the 2-way + verification process has been performed, the client has built a list of + all configuration entries that are out of spec. This list has two parts: + specified configuration that is incorrect (or missing) and unspecified + configuration that should be removed. + +* **Configuration Update** + + The client now attempts to update its configuration to match the + specification. Depending on options, changes may not (or only partially) + be performed. First, if extra configuration correction is enabled, + extra configuration can be removed. Then the remaining changes + are processed. The Bcfg2 client loops while progress is made in the + correction of these incorrect configuration entries. This loop results + in the client being able to accomplish all it will be able to during + one execution. Once all entries are fixed, or no progress is being + made, the loop terminates. Once all configuration changes that can be + performed have been, bundle dependencies are handled. Bundle groupings + result in two different behaviors. Contained entries are assumed + to be inter-dependent. To address this, the client re-verifies each + entry in any bundle containing an updates configuration entry. Also, + services contained in modified bundles are restarted. + +* **Statistics Upload** + + Once the reconfiguration process has concluded, the client reports + information back to the server about the actions it performed during the + reconfiguration process. Statistics function as a detailed return code + from the client. The server stores statistics information. Information + included in this statistics update includes (but is not limited to): + + * Overall client status (clean/dirty) + * List of modified configuration entries + * List of uncorrectable configuration entries + * List of unmanaged configuration entries + +Architecture Abstraction +------------------------ + +The Bcfg2 client internally supports the administrative tools available +on different architectures. For example, ``rpm`` and ``apt-get`` are +both supported, allowing operation of Debian, Redhat, SUSE, and Mandriva +systems. The client toolset is determined based on the availability of +client tools. The client includes a series of libraries which describe +how to interact with the system tools on a particular platform. + +Three of the libraries exist. There is a base set of functions, which +contain definitions describing how to perform POSIX operations. Support +for configuration files, directories, symlinks, hardlinks, etc., are +included here. Two other libraries subclass this one, providing support +for Debian and rpm-based systems. + +The Debian toolset includes support for apt-get and update-rc.d. These +tools provide the ability to install and remove packages, and to install +and remove services. + +The Redhat toolset includes support for rpm and chkconfig. Any other +platform that uses these tools can also use this toolset. Hence, all +of the other familiar rpm-based distributions can use this toolset +without issue. + +Other platforms can easily use the POSIX toolset, ignoring support for +packages or services. Alternatively, adding support for new toolsets +isn't difficult. Each toolset consists of about 125 lines of python code. + +The Bcfg2 Server +================ + +The Bcfg2 server is responsible for taking a network description and +turning it into a series of configuration specifications for particular +clients. It also manages probed data and tracks statistics for clients. + +The Bcfg2 server takes information from two sources when generating +client configuration specifications. The first is a pool of metadata that +describes clients as members of an aspect-based classing system. That is, +clients are defined in terms of aspects of their behavior. The other is +a file system repository that contains mappings from metadata to literal +configuration. These are combined to form the literal configuration +specifications for clients. + +The Configuration Specification Construction Process +---------------------------------------------------- + +As we described in the previous section, the client connects to the server +to request a configuration specification. The server uses the client's +metadata and the file system repository to build a specification that +is tailored for the client. This process consists of the following steps: + +* **Metadata Lookup** + + The server uses the client's IP address to initiate the metadata + lookup. This initial metadata consists of a (profile, image) tuple. If + the client already has metadata registered, then it is used. If not, + then default values are used and stored for future use. This metadata + tuple is expanded using some profile and class definitions also included + in the metadata. The end result of this process is metadata consisting + of hostname, profile, image, a list of classes, a list of attributes + and a list of bundles. + +* **Abstract Configuration Construction** + + Once the server has the client metadata, it is used to create + an abstract configuration. An abstract configuration contains + all of the configuration elements that will exist in the final + specification **without** any specifics. All entries will be typed + (i.e. the tagname will be one of Package, Path, Action, etc) and will + include a name. These configuration entries are grouped into bundles, + which document installation time interdependencies. + +* **Configuration Binding** + + The abstract configuration determines the structure of the client + configuration, however, it doesn't yet contain literal configuration + information. After the abstract configuration is created, each + configuration entry must be bound to a client-specific value. The Bcfg2 + server uses plugins to provide these client-specific bindings. The Bcfg2 + server core contains a dispatch table that describes which plugins can + handle requests of a particular type. The responsible plugin is located + for each entry. It is called, passing in the configuration entry and + the client's metadata. The behavior of plugins is explicitly undefined, + so as to allow maximum flexibility. The behaviours of the stock plugins + are documented elsewhere in this manual. Once this binding process + is completed, the server has a literal, client-specific configuration + specification. This specification is complete and comprehensive; the + client doesn't need to process it at all in order to use it. It also + represents the totality of the configuration specified for the client. + +The Literal Configuration Specification +======================================= + +Literal configuration specifications are served to clients by the +Bcfg2 server. This is a differentiating factor for Bcfg2; all other +major configuration management systems use a non-literal configuration +specification. That is, the clients receive a symbolic configuration that +they process to implement target states. We took the literal approach +for a few reasons: + +* A small list of configuration element types can be defined, each of + which can have a set of defined semantics. This allows the server to + have a well-formed model of client-side operations. Without a static + lexicon with defined semantics, this isn't possible. This allows the + server, for example, to record the update of a package as a coherent + event. +* Literal configurations do not require client-side processing. Removing + client-side processing reduces the critical footprint of the tool. + That is, the Bcfg2 client (and the tools it calls) need to be + functional, but the rest of the system can be in any state. Yet, + the client will receive a correct configuration. +* Having static, defined element semantics also requires that all + operations be defined and implemented in advance. The implementation + can maximize reliability and robustness. In more ad-hoc setups, these + operations aren't necessarily safely implemented. + +The Structure of Specifications +------------------------------- + +Configuration specifications contain some number of clauses. Two types +of clauses exist. Bundles are groups of inter-dependent configuration +entities. The purpose of bundles is to encode installation-time +dependencies such that all new configuration is properly activated +during reconfiguration operations. That is, if a daemon configuration +file is changed, its daemon should be restarted. Another example of +bundle usage is the reconfiguration of a software package. If a package +contains a default configuration file, but it gets overwritten by an +environment-specific one, then that updated configuration file should +survive package upgrade. The purpose of bundles is to describe services, +or reconfigured software packages. Independent clauses contain groups +of configuration entities that aren't related in any way. This provides a +convenient mechanism that can be used for bulk installations of software. + +Each of these clauses contains some number of configuration entities. A +number of configuration entities exist including Path, Package, Service, +etc. Each of these correspond to the obvious system item. Configuration +specifications can get quite large; many systems have specifications +that top one megabyte in size. An example of one is included in an +appendix. These configurations can be written by hand, or generated by +the server. + +Design Considerations +===================== + +This section will discuss several aspects of the design of Bcfg2, and the +particular use cases that motivated them. Initially, this will consist +of a discussion of the system metadata, and the intended usage model +for package indices as well. + +System Metadata +--------------- + +Bcfg2 system metadata describes the underlying patterns in system +configurations. It describes commonalities and differences between these +specifications in a rigorous way. The groups used by Bcfg2's metadata are +responsible for differentiating clients from one another, and building +collections of allocatable configuration. + +The Bcfg2 metadata system has been designed with several high-level +goals in mind. Flexibility and precision are paramount concerns; no +configuration should be undescribable using the constructs present in +the Bcfg2 repository. We have found (generally the hard way) that any +assumptions about the inherent simplicity of configuration patterns tend +to be wrong, so obscenely complex configurations must be representable, +even if these requirements seem illogical during the implementation. + +In particular, we wanted to streamline several operations that commonly +occurred in our environment. + +* Copying one node's profile to another node. + + In many environments, many nodes are instances of a common configuration + specification. They all have similar roles and software. In our + environment, desktop machines were the best example of this. Other than + strictly per-host configuration like SSH keys, all desktop machines + use a common configuration specification. This trivializes the process + of creating a new desktop machine. + +* Creating a specialized version of an existing profile. + + In environments with highly varied configurations, departmental + infrastructure being a good example, "another machine like X but with + extra software" is a common requirement. For this reason, it must be + trivially possible to inherit most of a configuration specification + from some more generic source, while being able to describe overriding + aspects in a convenient fashion. + +* Compose several pre-existing configuration aspects to create a new profile. + + The ability to compose configuration aspects allows the easy creation + of new profiles based on a series of predefined set of configuration + specification fragments. The end result is more agility in environments + where change is the norm. + + In order for a classing system to be comprehensive, it must be usable in + complex ways. The Bcfg2 metadata system has constructs that map cleanly + to first-order logic. This implies that any complex configuration + pattern can be represented (at all) by the metadata, as first-order + logic is provably comprehensive. (There is a discussion later in the + document describing the metadata system in detail, and showing how it + corresponds to first-order logic) + +These use cases motivate several of the design decisions that we +made. There must be a many to one correspondence between clients and +groups. Membership in a given profile group must imbue a client with +all of its configuration properties. + +Package Management +------------------ + +The interface provided in the Bcfg2 repository for package specification +was designed with automation in mind. The goal was to support an +append only interface to the repository, so that users do not need to +continuously re-write already existing bits of specification. diff --git a/doc/contents.txt b/doc/contents.txt index 80c4a9072..6b568547d 100644 --- a/doc/contents.txt +++ b/doc/contents.txt @@ -18,6 +18,7 @@ Welcome to Bcfg2's documentation! client/index gettinghelp bcfg2.conf-options + architecture development/index unsorted/index diff --git a/doc/index.txt b/doc/index.txt index 54c8176c2..c9252a379 100644 --- a/doc/index.txt +++ b/doc/index.txt @@ -51,6 +51,9 @@ client configuration state information. The design and implementation of the reporting system is described in a separate :ref:`page <server-reports-index>`. +A comprehensive description of the Bcfg2 Architecture (and the choices +behind the design) can be found at :ref:`architecture` + Server ^^^^^^ diff --git a/doc/server/configurationentries.txt b/doc/server/configurationentries.txt index d4a1407d2..4e4516106 100644 --- a/doc/server/configurationentries.txt +++ b/doc/server/configurationentries.txt @@ -2,14 +2,15 @@ .. _server-configurationentries: +===================== Configuration Entries ---------------------- +===================== This page describes the names and semantics of each of the configuration entries used by Bcfg2. -Common non-POSIX entries -^^^^^^^^^^^^^^^^^^^^^^^^ +Non-POSIX entries +================= +-------------+---------------------+-----------------------------+ | TagName | Description | Attributes | @@ -24,13 +25,16 @@ Common non-POSIX entries +-------------+---------------------+-----------------------------+ POSIX entries -^^^^^^^^^^^^^ +============= -In 1.0.0, there are new unified POSIX Path entries which prevent +Starting in 1.0.0, there are new unified POSIX Path entries which prevent inconsistent configuration specifications of multiple entries for a given path. The following table describes the various types available for new **Path** entries. +.. note:: A tool for converting old POSIX entries is available in the + bcfg2 source directory at tools/posixunified.py + +-------------+----------------------+-----------------+--------------------------+ | Type | Replacement/New | Description | Attributes | +=============+======================+=================+==========================+ @@ -60,3 +64,122 @@ path. The following table describes the various types available for new | symlink | Replaces SymLink | SymLinks | name, to | | | entries | | | +-------------+----------------------+-----------------+--------------------------+ + +Bound Entries +============= + +This feature is a mechanism to specify a full entry at once from +a bundle. Traditionally, entries are defined in two stages. First, +an abstract entry is defined in a bundle. This entry includes a type +(the XML tag) and a name attribute. Then this entry is bound for +a client, providing the appropriate instance of that entry for the +client. Specifying a bound entry short-circuits this process; the only +second stage processing on Bound entries is to remove the "Bound" prefix +from the element tag. The use of a bound entry allows the single stage +definition of a complete entry. Bound entries can be used for any type. + +Example: + +.. code-block:: xml + + <Bundle name='ntp'> + <BoundPackage name='ntp' type='deb' version='1:4.2.4p4+dfsg-3ubuntu2.1'/> + </Bundle> + +Fun and Profit using altsrc +=========================== + +Altsrc is a generic, bcfg2-server-side mechanism for performing +configuration entry name remapping for the purpose of data binding. + +Use Cases +--------- + +* Equivalent configuration entries on different architectures with + different names + +* Mapping entries with the same name to different bind results in a + configuration (two packages with the same name but different types) + +* A single configuration entry across multiple specifications + (multi-plugin, or multi-repo) + +Examples +-------- + +* Consider the case of /etc/hosts on linux and ``/etc/inet/hosts`` on + solaris. These files contain the same data in the same format, + and should typically be synchronized, however, exist in different + locations. Classically, one would need to create one entry for each in + Cfg or TCheetah and perform manual synchronization. Or, you could use + symlinks and pray. Altsrc is driven from the bundle side. For example: + + .. code-block:: xml + + <Bundle name='netinfo'> + <Group name='solaris'> + <Path name='/etc/inet/hosts' altsrc='/etc/hosts'/> + </Group> + <Group name='linux'> + <Path name='/etc/hosts'/> + </Group> + </Bundle> + + In this case, when a solaris host gets the 'netinfo' bundle, it will + get the first Path entry, which includes an altsrc parameter. This + will cause the server to bind the entry as if it were a Path + called ``/etc/hosts``. This configuration entry is still called + ``/etc/inet/hosts``, and is installed as such. + +* On encap systems, frequently multiple packages of the same name, but + of different types will exist. For example, there might be an openssl + encap package, and an openssl rpm package. This can be dealt with + using a bundle like: + + .. code-block:: xml + + <Bundle name='openssl'> + <Package name='openssl' altsrc='openssl-encap'/> + <Package name='openssl' altsrc='openssl-rpm'/> + </Bundle> + + This bundle will bind data for the packages "openssl-encap" and + "openssl-rpm", but will be delivered to the client with both packages + named "openssl" with different types. + +* Finally, consider the case where there exist complicated, but + completely independent specifications for the same configuration entry + but different groups of clients. The following bundle will allow the use + of two different TCheetah templates ``/etc/firewall-rules-external`` + and ``/etc/firewall-rules-internal`` for different clients based on + their group membership. + + .. code-block:: xml + + <Bundle name='firewall'> + ... + <Group name='conduit'> + <Path name='/etc/firewall-rules' altsrc='/etc/firewall-rules-external'/> + </Group> + <Group name='internal'> + <Path name='/etc/firewall-rules' altsrc='/etc/firewall-rules-internal'/> + </Group> + </Bundle> + +* Consider the case where a variety of files can be constructed by a + single template (TCheetah or TGenshi). It would be possible to copy + this template into the proper location for each file, but that requires + proper synchronization upon modification and knowing up front what + the files will all be called. Instead, the following bundle allows + the use of a single template for all proper config file instances. + + .. code-block:: xml + + <Bundle name='netconfig'> + <Path name='/etc/sysconfig/network-scripts/ifcfg-eth0' altsrc='/etc/ifcfg-template'/> + <Path name='/etc/sysconfig/network-scripts/ifcfg-eth1' altsrc='/etc/ifcfg-template'/> + <Path name='/etc/sysconfig/network-scripts/ifcfg-eth2' altsrc='/etc/ifcfg-template'/> + </Bundle> + + altsrc can be used as a parameter for any entry type, and can be used + in any structure, including Bundler and Base. diff --git a/doc/server/index.txt b/doc/server/index.txt index 17143dfe6..753d24fbe 100644 --- a/doc/server/index.txt +++ b/doc/server/index.txt @@ -11,8 +11,4 @@ Bcfg2 Server plugins/index reports/index - -.. toctree:: - :hidden: - configurationentries diff --git a/doc/server/plugins/generators/hostbase.txt b/doc/server/plugins/generators/hostbase.txt index 13e2ecabb..bf28e401d 100644 --- a/doc/server/plugins/generators/hostbase.txt +++ b/doc/server/plugins/generators/hostbase.txt @@ -13,9 +13,6 @@ plugin that generates dhcp, dns, and yp configuration files. Installation ============ -Overview --------- - Installation of Hostbase requires installation of a python module, configuration of database (mysql or postgres), and configuration of an Apache webserver with mod_python. Hostbase was developed using MySQL, @@ -208,20 +205,23 @@ source to your machine. When this command is run as root, it triggers the Hostbase to rebuild it's files, then runs the bcfg2 client on your local machine to grab the new configs. -Authentication -============== - -Edit Django settings --------------------- +NIS Authentication +================== Django allows for custom authentication backends to its login procedure. Hostbase has an NIS authentication backend that verifies a user to be in the unix group allowed to modify Hostbase. To enable this feature: - * first edit your ``Hostbase/settings.py`` file and uncomment the line ``'Hostbase.backends.NISBackend',`` in the list of ``AUTHENTICATION_BACKENDS`` - * enter the name of the unix group you want to give access to Hostbase in the ``AUTHORIZED_GROUP`` variable - * in your ``Hostbase/hostbase/views.py`` file at the very bottom, uncomment the block(s) of lines that give you the desired level of access + +* first edit your ``Hostbase/settings.py`` file and uncomment + the line **Hostbase.backends.NISBackend** in the list of + *AUTHENTICATION_BACKENDS* +* enter the name of the unix group you want to give access to Hostbase + in the *AUTHORIZED_GROUP* variable +* in your ``Hostbase/hostbase/views.py`` file at the very bottom, + uncomment the block(s) of lines that give you the desired level + of access Hostbase will now direct the user to a login page if he or she is not authorized to view a certain page. Users should log in with their diff --git a/doc/server/snapshots/index.txt b/doc/server/snapshots/index.txt new file mode 100644 index 000000000..a961aa167 --- /dev/null +++ b/doc/server/snapshots/index.txt @@ -0,0 +1,22 @@ +.. -*- mode: rst -*- + +.. _server-snapshots-index: + +Bcfg2 Snapshots +=============== + +Bcfg2's reporting system is its killer feature. It allows administrators +to gain a broad understanding of the configuration state of their entire +environment. It summarizes + +* Configuration changes and when they were made +* Discrepancies between the specification and current client states + + * Clients can be grouped by misconfiguration type + +* Configuration entries that are not specified +* Overall client summaries according to these types + +There are two systems, the old system, which builds static reports based +on a series of XSLT stylesheets and a new dynamic reporting system that +uses django and a database backend. diff --git a/doc/unsorted/altsrc.txt b/doc/unsorted/altsrc.txt deleted file mode 100644 index 251781d28..000000000 --- a/doc/unsorted/altsrc.txt +++ /dev/null @@ -1,102 +0,0 @@ -.. -*- mode: rst -*- - -.. _unsorted-altsrc: - -=========================== -Fun and Profit using altsrc -=========================== - -Altsrc is a generic, bcfg2-server-side mechanism for performing -configuration entry name remapping for the purpose of data binding. - -Use Cases -========= - -* Equivalent configuration entries on different architectures with - different names - -* Mapping entries with the same name to different bind results in a - configuration (two packages with the same name but different types) - -* A single configuration entry across multiple specifications - (multi-plugin, or multi-repo) - -Examples -======== - -* Consider the case of /etc/hosts on linux and ``/etc/inet/hosts`` on - solaris. These files contain the same data in the same format, - and should typically be synchronized, however, exist in different - locations. Classically, one would need to create one entry for each in - Cfg or TCheetah and perform manual synchronization. Or, you could use - symlinks and pray. Altsrc is driven from the bundle side. For example: - - .. code-block:: xml - - <Bundle name='netinfo'> - <Group name='solaris'> - <Path name='/etc/inet/hosts' altsrc='/etc/hosts'/> - </Group> - <Group name='linux'> - <Path name='/etc/hosts'/> - </Group> - </Bundle> - - In this case, when a solaris host gets the 'netinfo' bundle, it will - get the first Path entry, which includes an altsrc parameter. This - will cause the server to bind the entry as if it were a Path - called ``/etc/hosts``. This configuration entry is still called - ``/etc/inet/hosts``, and is installed as such. - -* On encap systems, frequently multiple packages of the same name, but - of different types will exist. For example, there might be an openssl - encap package, and an openssl rpm package. This can be dealt with - using a bundle like: - - .. code-block:: xml - - <Bundle name='openssl'> - <Package name='openssl' altsrc='openssl-encap'/> - <Package name='openssl' altsrc='openssl-rpm'/> - </Bundle> - - This bundle will bind data for the packages "openssl-encap" and - "openssl-rpm", but will be delivered to the client with both packages - named "openssl" with different types. - -* Finally, consider the case where there exist complicated, but - completely independent specifications for the same configuration entry - but different groups of clients. The following bundle will allow the use - of two different TCheetah templates ``/etc/firewall-rules-external`` - and ``/etc/firewall-rules-internal`` for different clients based on - their group membership. - - .. code-block:: xml - - <Bundle name='firewall'> - ... - <Group name='conduit'> - <Path name='/etc/firewall-rules' altsrc='/etc/firewall-rules-external'/> - </Group> - <Group name='internal'> - <Path name='/etc/firewall-rules' altsrc='/etc/firewall-rules-internal'/> - </Group> - </Bundle> - -* Consider the case where a variety of files can be constructed by a - single template (TCheetah or TGenshi). It would be possible to copy - this template into the proper location for each file, but that requires - proper synchronization upon modification and knowing up front what - the files will all be called. Instead, the following bundle allows - the use of a single template for all proper config file instances. - - .. code-block:: xml - - <Bundle name='netconfig'> - <Path name='/etc/sysconfig/network-scripts/ifcfg-eth0' altsrc='/etc/ifcfg-template'/> - <Path name='/etc/sysconfig/network-scripts/ifcfg-eth1' altsrc='/etc/ifcfg-template'/> - <Path name='/etc/sysconfig/network-scripts/ifcfg-eth2' altsrc='/etc/ifcfg-template'/> - </Bundle> - - altsrc can be used as a parameter for any entry type, and can be used - in any structure, including Bundler and Base. diff --git a/doc/unsorted/annotated_examples.txt b/doc/unsorted/annotated_examples.txt deleted file mode 100644 index b5b7e6b4e..000000000 --- a/doc/unsorted/annotated_examples.txt +++ /dev/null @@ -1,191 +0,0 @@ -.. -*- mode: rst -*- - -.. _unsorted-annotated_examples: - -================== -Annotated Examples -================== - -ntp example -=========== - -Author: Jason Pepas - -Here is a series of example configurations for bcfg2, each introducing -another layer of functionality. - -* After each change, run 'bcfg-repo-validate -v'. -* Run the server with 'bcfg2-server -v'. -* Update the client with 'bcfg2 -v -d -n'. (will not actually make client changes) - -package only ------------- - -Our example starts with the bare minimum configuration setup. We have -a client, a profile group, a list of packages, and a base configuration. - -:: - - # cat Metadata/clients.xml - <Clients version='3.0'> - <Client profile='fedora' pingable='N' pingtime='0' name='foo.bar.com'/> - </Clients> - - # cat Metadata/groups.xml - <Groups version='3.0'> - <Group profile='true' name='fedora' toolset='rh'/> - </Groups> - - # cat Base/base.xml - <Base> - <Group name='fedora'> - <Package name='ntp'/> - </Group> - </Base> - - # cat Pkgmgr/packages.xml - <PackageList type='rpm' priority='0'> - <Package name='ntp' version='4.2.0.a.20050816-11.FC5'/> - </PackageList> - -add service ------------ - -Configure the service, and add it to the base.:: - - # cat Svcmgr/services.xml - <Services priority='0'> - <Service name='ntpd' status='on'/> - </Services> - - # cat Base/base.xml - <Base> - <Group name='fedora'> - <Package name='ntp'/> - <Service name='ntpd'/> - </Group> - </Base> - -add config file ---------------- - -Setup an etc directory structure, and add it to the base.:: - - # cat Cfg/etc/ntp.conf/ntp.conf - server ntp1.utexas.edu - - # cat Base/base.xml - <Base> - <Group name='fedora'> - <Package name='ntp'/> - <Service name='ntpd'/> - <Path name='/etc/ntp.conf'/> - </Group> - </Base> - -create a bundle ---------------- - -The above configuration layout works fine for a single service, but -that method of organization would quickly become a nightmare as you -approach the number of packages, services, and config files required to -represent a fully configured host. Bundles allow the grouping of related -configuration entries that are used to provide a single service. This -is done for several reasons: - -* Grouping related things in one place makes it easier to add those - entries for a multiple groups of clients -* Grouping entries into bundles makes their validation occur - collectively. This means that config files can override the - contents of packages. Also, config files are rechecked after - packages are upgraded, so that they can be repaired if the package - install clobbered them. -* Services associated with a bundle get restarted whenever any - entity in that bundle is modified. This ensures that new - configuration files and software are used after installation. - -The config file, package, and service are really all related components -describing the idea of an ntp client, so they should be logically -grouped together. We use a bundle to accomplish this.:: - - # cat Bundler/ntp.xml - <Bundle name='ntp' version='2.0'> - <Package name='ntp'/> - <Service name='ntpd'/> - <Path name='/etc/ntp.conf'/> - </Bundle> - -After this bundle is created, it must be associated with a group (or -groups). Add a bundle child element to the group(s) which should install -this bundle.:: - - - # cat Metadata/groups.xml - <Groups> - ... - <Group name='fedora'> - <Bundle name='ntp'/> - </Group> - ... - </Groups> - -Once this bundle is created, a client reconfigure will install these -entries. If any are modified, then the ntpd service will be -restarted. If you only want ntp configurations to be updated (and -nothing else), the bcfg2 client can be run with a -b <bundle name> -option that will only update entries in the specified bundle. - -mysql example -============= - -Author: Patrick Ruckstuhl - -I had some time ago to continue with putting my configuration into -bcfg2 and maybe this helps someone else. - -I added a new bundle: - -.. code-block: xml - - <Bundle name="mysql-server" version="3.0"> - <Path name="/root/bcfg2-install/mysql/users.sh"/> - <Path name="/root/bcfg2-install/mysql/users.sql"/> - <PostInstall name="/root/bcfg2-install/mysql/users.sh"/> - <Package name="mysql-server-4.1"/> - <Service name="mysql"/> - </Bundle> - -The `users.sh` script looks like this: - -.. code-block: sh - - #!/bin/sh - - mysql --defaults-extra-file=/etc/mysql/debian.cnf mysql \ - < /root/bcfg2-install/mysql/users.sql - -On debian there is a user account in ``/etc/mysql/debian.cnf`` -automatically created, but you could also (manually) create a user in -the database that has enough permissions and add the login information -in a file yourself. This file looks like this:: - - [client] - host = localhost - user = debian-sys-maint - password = XXXXXXXXXX - -The ``users.sql`` looks like this:: - - DELETE FROM db; - INSERT INTO db VALUES ('localhost', 'phpmyadmin', 'pma', 'Y', 'Y', - 'Y', 'Y', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N'); - - DELETE FROM user WHERE User <> 'debian-sys-maint'; - INSERT INTO user VALUES ('localhost', 'root', 'XXXXXXXXXXX', 'Y', 'Y', - 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', - 'Y', 'Y', 'Y', 'Y', 'Y', '', '', '', '', 0, 0, 0); - INSERT INTO user VALUES ('localhost', 'pma', '', 'N', 'N', 'N', 'N', - 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', 'N', - 'N', 'N', 'N', '', '', '', '', 0, 0, 0); - - FLUSH PRIVILEGES; diff --git a/doc/unsorted/boundentry.txt b/doc/unsorted/boundentry.txt deleted file mode 100644 index 6dfa4b9bd..000000000 --- a/doc/unsorted/boundentry.txt +++ /dev/null @@ -1,15 +0,0 @@ -.. -*- mode: rst -*- - -.. _unsorted-boundentry: - -============= -Bound Entries -============= - -This feature is a mechanism to specify a full entry at once from a bundle. Traditionally, entries are defined in two stages. First, an abstract entry is defined in a bundle. This entry includes a type (the XML tag) and a name attribute. Then this entry is bound for a client, providing the appropriate instance of that entry for the client. Specifying a bound entry short-circuits this process; the only second stage processing on Bound entries is to remove the "Bound" prefix from the element tag. The use of a bound entry allows the single stage definition of a complete entry. Bound entries can be used for any type. - -.. code-block:: xml - - <Bundle name='ntp'> - <BoundPackage name='ntp' type='deb' version='1:4.2.4p4+dfsg-3ubuntu2.1'/> - </Bundle> diff --git a/doc/unsorted/contributors.txt b/doc/unsorted/contributors.txt deleted file mode 100644 index b9dd7cd77..000000000 --- a/doc/unsorted/contributors.txt +++ /dev/null @@ -1,32 +0,0 @@ -.. -*- mode: rst -*- - -.. _unsorted-contributors: - -============ -Contributors -============ - -A large number of people have contributed to Bcfg2. - -* Narayan Desai - wrote and designed much of the bcfg1/2 -* Andrew Lusk - early development and hardening -* Brian Pellin - early development and hardening -* Rick Bradshaw -* Cory Lueninghoener - Bugfixes and change management code -* Scott Matott -* Ti Leggett - ebuild packaging and bugfixes, RPM packaging -* Joey Hagedorn - Mac OS X support in bcfg1, launchd support in Bcfg2, and both versions of the reporting system -* Zach Lowry - Solaris support and general hardening -* Jos Catnook - Bugfixes -* Jason Pepas - Package manager source generation scripts -* Patrick Ruckstuhl - Templating bugfixes -* Chris Vuletich - statistics annotations and client features -* Ken Raffenetti - Hostbase -* Sami Haahtinen - Debian Packaging -* Danny Clark - Encap Packaging -* Michael Jinks - Gentoo support -* David Dahl - Hostbase -* Pedro Flores - Reporting system design help -* David Strauss - CentOS, RHEL, Yum, and Bazaar VCS support - -The entire MCS systems team has provided invaluable help in the design process and refinement of the user interface. In particular, Gene Rackow and Sandra Bittner have provided great assistance throughout this project. Philip Steinbachs provided detailed feedback as an early external user. diff --git a/doc/unsorted/doc_architecture.txt b/doc/unsorted/doc_architecture.txt deleted file mode 100644 index 0aca0ef10..000000000 --- a/doc/unsorted/doc_architecture.txt +++ /dev/null @@ -1,128 +0,0 @@ -.. -*- mode: rst -*- - -.. _unsorted-doc_architecture: - -================== -Bcfg2 Architecture -================== - -Bcfg2 is based on a client-server architecture. The client is responsible for interpreting (but not processing) the configuration served by the server. This configuration is literal, so no local process is required. After completion of the configuration process, the client uploads a set of statistics to the server. This section will describe the goals and then the architecture motivated by it. - -Goals -===== - -* '''Model configurations using declarative semantics.'''[[BR]] - Declarative semantics maximize the utility of configuration management tools; they provide the most flexibility for the tool to determine the right course of action in any given situation. This means that users can focus on the task of describing the desired configuration, while leaving the task of transitioning clients states to the tool. -* '''Configuration descriptions should be comprehensive.'''[[BR]] - This means that configurations served to the client should be sufficient to reproduce all desired functionality. This assumption allows the use of heuristics to detect extra configuration, aiding in reliable, comprehensive configuration definitions. -* '''Provide a flexible approach to user interactions.'''[[BR]] - Most configuration management systems take a rigid approach to user interactions; that is, either the client system is always correct, or the central system is. This means that users are forced into an overly proscribed model where the system asserts where correct data is. Configuration data modification is frequently undertaken on both the configuration server and clients. Hence, the existence of a single canonical data location can easily pose a problem during normal tool use. Bcfg2 takes a different approach. - -The default assumption is that data on the server is correct, however, the client has option to run in another mode where local changes are catalogued for server-side integration. If the Bcfg2 client is run in dry run mode, it can help to reconcile differences between current client state and the configuration described on the server. -The Bcfg2 client also searches for extra configuration; that is, configuration that is not specified by the configuration description. When extra configuration is found, either configuration has been removed from the configuration description on the server, or manual configuration has occurred on the client. Options related to two-way verification and removal are useful for configuration reconciliation when interactive access is used. - -* Plugins and administrative applications. -* Incremental operations. - -The Bcfg2 Client -================ - -The Bcfg2 client performs all client configuration or reconfiguration operations. It renders a declarative configuration specification, provided by the Bcfg2 server, into a set of configuration operations which will, if executed, attempt to change the client's state into that described by the configuration specification. Conceptually, the Bcfg2 client serves to isolate the Bcfg2 server and specification from the imperative operations required to implement configuration changes. - -This isolation allows declarative specifications to be manipulated symbolically on the server, without needing to understand the properties of the underlying system tools. In this way, the Bcfg2 client acts as a sort of expert system that "knows" how to implement declarative configuration changes. - -The operation of the Bcfg2 client is intended to be as simple as possible. The normal configuration process consists of four main steps: - -* Probe Execution - During the probe execution stage, the client connects to the server and downloads a series of probes to execute. These probes reveal local facts to the Bcfg2 server. For example, a probe could discover the type of video card in a system. The Bcfg2 client returns this data to the server, where it can influence the client configuration generation process. -* Configuration Download and Inventory - The Bcfg2 client now downloads a configuration specification from the Bcfg2 server. The configuration describes the complete target state of the machine. That is, all aspects of client configuration should be represented in this specification. For example, all software packages and services should be represented in the configuration specification. The client now performs a local system inventory. This process consists of verifying each entry present in the configuration specification. After this check is completed, heuristic checks for configuration not included in the configuration specification. We refer to this inventory process as 2-way validation, as first we verify that the client contains all configuration that is included in the specification, then we check if the client has any extra configuration that isn't present. This provides a fairly rigorous notion of client configuration congruence. Once the 2-way verification process has been performed, the client has built a list of all configuration entries that are out of spec. This list has two parts: specified configuration that is incorrect (or missing) and unspecified configuration that should be removed. -* Configuration Update - The client now attempts to update its configuration to match the specification. Depending on options, changes may not (or only partially) be performed. First, if extra configuration correction is enabled, extra configuration can be removed. Then the remaining changes are processed. The Bcfg2 client loops while progress is made in the correction of these incorrect configuration entries. This loop results in the client being able to accomplish all it will be able to during one execution. Once all entries are fixed, or no progress is being made, the loop terminates.Once all configuration changes that can be performed have been, bundle dependencies are handled. Bundle groupings result in two different behaviors. Contained entries are assumed to be inter-dependant. To address this, the client re-verifies each entry in any bundle containing an updates configuration entry. Also, services contained in modified bundles are restarted. -* Statistics Upload - Once the reconfiguration process has concluded, the client reports information back to the server about the actions it performed during the reconfiguration process. Statistics function as a detailed return code from the client. The server stores statistics information. Information included in this statistics update includes (but is not limited to): - - * Overall client status (clean/dirty) - * List of modified configuration entries - * List of uncorrectable configuration entries - -Architecture Abstraction ------------------------- - -The Bcfg2 client internally supports the administrative tools available on different architectures. For example, rpm and apt-get are both supported, allowing operation of Debian, Redhat, SUSE, and Mandriva systems. The client toolset is specified in the configuration specification. The client merely includes a series of libraries which describe how to interact with the system tools on a particular platform. - -Three of the libraries exist. There is a base set of functions, which contain definitions describing how to perform POSIX operations. Support for configuration files, directories, and symlinks are included here. Two other libraries subclass this one, providing support for Debian and rpm-based systems. - -The Debian toolset includes support for apt-get and update-rc.d. These tools provide the ability to install and remove packages, and to install and remove services. - -The Redhat toolset includes support for rpm and chkconfig. Any other platform that uses these tools can also use this toolset. Hence, all of the other familiar rpm-based distributions can use this toolset without issue. - -Other platforms can easily use the POSIX toolset, ignoring support for packages or services. Alternatively, adding support for new toolsets isn't difficult. Each toolset consists of about 125 lines of python code. - - -The Bcfg2 Server -================ - -The Bcfg2 server is responsible for taking a network description and turning it into a series of configuration specifications for particular clients. It also manages probed data and tracks statistics for clients. - -The Bcfg2 server takes information from two sources when generating client configuration specifications. The first is a pool of metadata that describes clients as members of an aspect-based classing system. That is, clients are defined in terms of aspects of their behavior. The other is a file system repository that contains mappings from metadata to literal configuration. These are combined to form the literal configuration specifications for clients. - - -The Configuration Specification Construction Process ----------------------------------------------------- - -As we described in the previous section, the client connects to the server to request a configuration specification. The server uses the client's metadata and the file system repository to build a specification that is tailored for the client. This process consists of the following steps: - -* Metadata Lookup - The server uses the client's IP address to initiate the metadata lookup. This initial metadata consists of a (profile, image) tuple. If the client already has metadata registered, then it is used. If not, then default values are used and stored for future use. This metadata tuple is expanded using some profile and class definitions also included in the metadata. The end result of this process is metadata consisting of hostname, profile, image, a list of classes, a list of attributes and a list of bundles. -* Abstract Configuration Construction - Once the server has the client metadata, it is used to create an abstract configuration. An abstract configuration contains all of the configuration elements that will exist in the final specification without any specifics. All entries will be typed (ie the tagname will be one of Package, !ConfigurationFile, Service, !Symlink, or Directory) and will include a name. These configuration entries are grouped into bundles, which document installation time interdependencies. -* Configuration Binding - The abstract configuration determines the structure of the client configuration, however, it doesn't contain literal configuration information. After the abstract configuration is created, each configuration entry must be bound to a client-specific value. The Bcfg2 server uses plugins to provide these client-specific bindings. The Bcfg2 server core contains a dispatch table that describes which plugins can handle requests of a particular type. The responsible plugin is located for each entry. It is called, passing in the configuration entry and the client's metadata. The behavior of plugins is explicitly undefined, so as to allow maximum flexibility. The behaviours of the stock plugins are documented elsewhere in this manual. Once this binding process is completed, the server has a literal, client-specific configuration specification. This specification is complete and comprehensive; the client doesn't need to process it at all in order to use it. It also represents the totality of the configuration specified for the client. - -The Literal Configuration Specification -======================================= - -Literal configuration specifications are served to clients by the Bcfg2 server. This is a differentiating factor for Bcfg2; all other major configuration management systems use a non-literal configuration specification. That is, the clients receive a symbolic configuration that they process to implement target states. We took the literal approach for a few reasons: - -* A small list of configuration element types can be defined, each of which can have a set of defined semantics. This allows the server to have a well-formed model of client-side operations. Without a static lexicon with defined semantics, this isn't possible. This allows the server, for example, to record the update of a package as a coherent event. -* Literal configurations do not require client-side processing. Removing client-side processing reduces the critical footprint of the tool. That is, the Bcfg2 client (and the tools it calls) need to be functional, but the rest of the system can be in any state. Yet, the client will receive a correct configuration. -* Having static, defined element semantics also requires that all operations be defined and implemented in advance. The implementation can maximize reliability and robustness. In more ad-hoc setups, these operations aren't necessarily safely implemented. - -The Structure of Specifications -------------------------------- - -Configuration specifications contain some number of clauses. Two types of clauses exist. Bundles are groups of inter-dependent configuration entities. The purpose of bundles is to encode installation-time dependencies such that all new configuration is properly activated during reconfiguration operations. T hat is, if a daemon configuration file is changed, its daemon should be restarted. Another example of bundle usage is the reconfiguration of a software package. If a package contains a default configuration file, but it gets overwritten by an environment-specific one, then that updated configuration file should survive package upgrade. The purpose of bundles is to describe services, or reconfigured software packages. Independent clauses contains groups of configuration entities that aren't related in any way. This provides a convenient mechanism that can be used for bulk installations of software. - -Each of these clauses contains some number of configuration entities. Five types of configuration entities exist: !ConfigurationFile, Package, !SymLink, Directory, and Service. Each of these correspond to the obvious system item. Configuration specifications can get quite large; many systems have specifications that top one megabyte in size. An example of one is included in an appendix. These configurations can be written by hand, or generated by the server. The easiest way to start using Bcfg2 is to write small static configurations for clients. Once configurations get larger, this process gets unwieldy; at this point, using the server makes more sense. - -Design Considerations -===================== - -This section will discuss several aspects of the design of bcfg2, and the particular use cases that motivated them. Initially, this will consist of a discussion of the system metadata, and the intended usage model for package indices as well. - -System Metadata ---------------- - -Bcfg2 system metadata describes the underlying patterns in system configurations. It describes commonalities and differences between these specifications in a rigorous way. The groups used by bcfg2's metadata are responsible for differentiating clients from one another, and building collections of allocatable configuration. - -The Bcfg2 metadata system has been designed with several high-level goals in mind. Flexibility and precision are paramount concerns; no configuration should be undescribable using the constructs present in the bcfg2 repository. We have found (generally the hard way) that any assumptions about the inherent simplicity of configuration patterns tend to be wrong, so obscenely complex configurations must be representable, even if these requirements seem illogical during the implementation. - -In particular, we wanted to streamline several operations that commonly occurred in our environment. - -* Copying one node's profile to another node. - In many environments, many nodes are instances of a common configuration specification. They all have similar roles and software. In our environment, desktop machines were the best example of this. Other than strictly per-host configuration like SSH keys, all desktop machines use a common configuration specification. This trivializes the process of creating a new desktop machine. -* Creating a specialized version of a currently existing profile. - In environments with highly varied configurations, departmental infrastructure being a good example, "another machine like X but with extra software" is a common requirement. For this reason, it must be trivially possible to inherit most of a configuration specification from some more generic source, while being able to describe overriding aspects in a convenient fashion. -* 3. Compose several pre-existing configuration aspects to create a new profile. - The ability to compose configuration aspects allows the easy creation of new profiles based on a series of predefined set of configuration specification fragments. The end result is more agility in environments where change is the norm. - - In order for a classing system to be comprehensive, it must be usable in complex ways. The Bcfg2 metadata system has constructs that map cleanly to first-order logic. This implies that any complex configuration pattern can be represented (at all) by the metadata, as first-order logic is provably comprehensive. (There is a discussion later in the document describing the metadata system in detail, and showing how it corresponds to first-order logic) - -These use cases motivate several of the design decisions that we made: -* There must be a many to one correspondence between clients and groups. Membership in a given profile group must imbue a client with all of its configuration properties. - -Package Management ------------------- - -The interface provided in the bcfg2 repository for package specification was designed with automation in mind. The goal was to support an append only interface to the repository, so that users do not need to continuously re-write already existing bits of specification. diff --git a/doc/unsorted/gentoo.txt b/doc/unsorted/gentoo.txt index 2375fcb17..015a19687 100644 --- a/doc/unsorted/gentoo.txt +++ b/doc/unsorted/gentoo.txt @@ -6,46 +6,61 @@ Gentoo ====== -''This document tries to lay out anything Gentoo-specific that you need to know in order to use bcfg2. Mostly that has to do with getting it to cooperate with the various pieces of Portage. Services, all things POSIX, and just about anything else that bcfg2 does will work the same on Gentoo as on any other distribution. bcfg2 is new on Gentoo; please let the list know if you find errors or omissions.'' +This document tries to lay out anything Gentoo-specific that you need +to know in order to use bcfg2. Mostly that has to do with getting it +to cooperate with the various pieces of Portage. Services, all things +POSIX, and just about anything else that bcfg2 does will work the same +on Gentoo as on any other distribution. bcfg2 is new on Gentoo; please +let the list know if you find errors or omissions. Installing bcfg2 ================ -Early in July 2008, bcfg2 was added to the Gentoo portage tree. So far it's only keyworded for ~x86, but we hope to see it soon in the amd64 and x64-solaris ports. If you're using Gentoo on some other architecture, it should still work provided that you have a reasonably up to date Python; try adding `app-admin/bcfg2 ~*` to your `/etc/portage/package.keywords` file. +Early in July 2008, bcfg2 was added to the Gentoo portage tree. So far +it's only keyworded for ~x86, but we hope to see it soon in the amd64 and +x64-solaris ports. If you're using Gentoo on some other architecture, it +should still work provided that you have a reasonably up to date Python; +try adding `app-admin/bcfg2 ~*` to your `/etc/portage/package.keywords` +file. -If you don’t use emerge to install bcfg2, you’ll want to make sure you have all the prerequisites installed first. For a server, you’ll need: +If you don’t use portage to install bcfg2, you’ll want to make sure you +have all the prerequisites installed first. For a server, you’ll need: -* `app-admin/gamin` or `app-admin/fam`; -* `dev-python/pyopenssl`; -* `dev-python/lxml` +* ``app-admin/gamin`` or ``app-admin/fam`` +* ``dev-python/lxml`` Clients will need at least: -* `app-portage/gentoolkit`; -* `dev-python/lxml` or `dev-python/elementtree` (if you’re running python-2.4 or below); +* ``app-portage/gentoolkit`` Package Repository ================== You’ll need (to make) at least one archive of binary packages. The -Portage driver calls `emerge` with the `–getbinpkgonly` option. See -:manpage:`make.conf(5)` and :manpage:`emerge(1)` manpages, specifically the `PORTAGE_BINHOST` environment variable. +Portage driver calls ``emerge`` with the ``–getbinpkgonly`` option. See +:manpage:`make.conf(5)` and :manpage:`emerge(1)` manpages, specifically +the *PORTAGE_BINHOST* environment variable. Time Saver: quickpkg -------------------- -If you have a standing Gentoo machine that you want to preserve or propagate, you can generate a complete package archive based on the present state of the system by using the quickpkg utility. For example: +If you have a standing Gentoo machine that you want to preserve or +propagate, you can generate a complete package archive based on the +present state of the system by using the quickpkg utility. For example: .. code-block:: sh for pkg in `equery -q l` ; do quickpkg "=$pkg" ; done -…will leave you with a complete archive of all the packages on your system in `/usr/portage/packages/All`, which you can then move to your ftp server. +…will leave you with a complete archive of all the packages on your +system in ``/usr/portage/packages/All``, which you can then move to your +ftp server. Cataloging Packages In Your Repository -------------------------------------- -Once you have a set of packages, you’ll need to create a catalog for them in /var/lib/bcfg2/Pkgmgr. Here’s a template: +Once you have a set of packages, you will need to create a catalog for +them in ``/var/lib/bcfg2/Pkgmgr``. Here’s a template: .. code-block:: xml @@ -67,11 +82,17 @@ Once you have a set of packages, you’ll need to create a catalog for them in / </Group> </PackageList> -The `<Group>` name (in our example, “gentoo-200701-vmware”) should be included by any host which will draw its packages from this list. Our collection of packages for this class of machines is at the listed URI, and we only have one collection of packages for this batch of machines so in our case the `priority` doesn’t really matter, we’ve set it to `0`. +The `<Group>` name (in our example, “gentoo-200701-vmware”) should +be included by any host which will draw its packages from this list. Our +collection of packages for this class of machines is at the listed URI, +and we only have one collection of packages for this batch of machines so +in our case the `priority` doesn’t really matter, we’ve set it to `0`. Notice that package name fields are in `CAT/TITLE` format. -Here’s a hack which will generate a list of Package lines from a system’s database of installed packages, especially useful in conjunction with the `quickpkg` example above: +Here is a hack which will generate a list of Package lines from +a system’s database of installed packages, especially useful in +conjunction with the `quickpkg` example above: .. code-block:: sh @@ -85,9 +106,16 @@ Here’s a hack which will generate a list of Package lines from a system’s da Configuring Client Machines =========================== -Set up `/etc/bcfg2.conf` the way you would for any other bcfg2 client. +Set up ``/etc/bcfg2.conf`` the way you would for any other bcfg2 client. -In `make.conf`, set `PORTAGE_BINHOST` to point to the URI of your package repository. You may want to create versions of `make.conf` for each package repository you maintain, with appropriate `PORTAGE_BINHOST` URI's in each, and associated with that package archive's group under `Cfg` -- for example, we have `Cfg/etc/make.conf/make.conf.G99_gentoo-200701-vmware`. If a client host switches groups, and the new group needs a different set of packages, everything should just fall into place. +In ``make.conf``, set *PORTAGE_BINHOST* to point to the URI of +your package repository. You may want to create versions of +``make.conf`` for each package repository you maintain, with +appropriate *PORTAGE_BINHOST* URI's in each, and associated with +that package archive's group under ``Cfg`` -- for example, we have +``Cfg/etc/make.conf/make.conf.G99_gentoo-200701-vmware``. If a client +host switches groups, and the new group needs a different set of packages, +everything should just fall into place. Pitfalls ======== @@ -116,13 +144,34 @@ leading to noise like this:: !!! /etc/ssh/ssh_config has incorrect md5sum * 62 out of 66 files good -''We can ignore the lines for `ssh_config` and `sshd_config`; those will be caught by bcfg2 as registered config files and handled appropriately.'' - -Because bcfg2 relies on the client system’s native package reporting tool to judge the state of installed packages, complaints like these about trivial or intractable verification failures can trigger unnecessary bundle reinstalls when the bcfg2 client runs. bcfg2 will catch on after a pass or two that the situation isn’t getting any better with repeated package installs, stop trying, and list those packages as “bad” in the client system’s statistics. - -Aside from filing bugs with the Gentoo package maintainers, your narrator has been unable to come up with a good approach to this. Maybe write a series of `Rules` definitions according to what the package database thinks it should find, and/or stage copies of affected files under `Cfg`, and associate those rules and files with the affected package in a bundle? Annoying but possibly necessary if you want your stats file to look good. +We can ignore the lines for ``ssh_config`` and ``sshd_config``; those will +be caught by bcfg2 as registered config files and handled appropriately. + +Because bcfg2 relies on the client system’s native package reporting +tool to judge the state of installed packages, complaints like these +about trivial or intractable verification failures can trigger unnecessary +bundle reinstalls when the bcfg2 client runs. bcfg2 will catch on after a +pass or two that the situation isn’t getting any better with repeated +package installs, stop trying, and list those packages as “bad” in +the client system’s statistics. + +Aside from filing bugs with the Gentoo package maintainers, your narrator +has been unable to come up with a good approach to this. Maybe write a +series of ``Rules`` definitions according to what the package database +thinks it should find, and/or stage copies of affected files under +``Cfg``, and associate those rules and files with the affected package in +a bundle? Annoying but possibly necessary if you want your stats file +to look good. /boot ----- -Gentoo as well as some other distros recommend leaving `/boot` unmounted during normal runtime. This can lead to trouble during verification and package installation, for example when `/boot/grub/grub.conf` turns up missing. The simplest way around this might just be to ensure that `/boot` is mounted whenever you run bcfg2, possibly wrapping bcfg2 in a script for the purpose. I’ve also thought about adding `Action` clauses to bundles for grub and our kernel packages, which would mount `/boot` before the bundle installs and unmount it afterward, but this doesn’t get around the problem of those packages flunking verification. +Gentoo as well as some other distros recommend leaving ``/boot`` unmounted +during normal runtime. This can lead to trouble during verification and +package installation, for example when ``/boot/grub/grub.conf`` turns +up missing. The simplest way around this might just be to ensure that +``/boot`` is mounted whenever you run bcfg2, possibly wrapping bcfg2 +in a script for the purpose. I’ve also thought about adding *Action* +clauses to bundles for grub and our kernel packages, which would mount +``/boot`` before the bundle installs and unmount it afterward, but this +doesn’t get around the problem of those packages flunking verification. diff --git a/doc/unsorted/groupnames.txt b/doc/unsorted/groupnames.txt deleted file mode 100644 index 7df223387..000000000 --- a/doc/unsorted/groupnames.txt +++ /dev/null @@ -1,17 +0,0 @@ -.. -*- mode: rst -*- - -.. _unsorted-groupnames: - -=========== -Group Names -=========== - -This pages describes a set of common group names. The goal is to build a small set of common group names that can be used across installations. With any luck, this will increase portability of repository contents. Use of these names is, and will continue to be entirely optional. None of these groups are profile groups. This will result in a highly functional example repository, hosted in [source:/trunk/repository svn] - -|| '''Group Name''' || '''Category''' || '''Semantic Content''' || -|| linux || os || Clients running the Linux kernel || -|| solaris || os || Clients running the Solaris kernel || - -The following image describes the base compatibility OS groups for bcfg2. This file is available from [source:/trunk/repository/Metadata/bcfg2-os.xml]. - -.. image:: viz.png diff --git a/doc/unsorted/mailinglist.txt b/doc/unsorted/mailinglist.txt index d7cf8a8ca..8f49b2706 100644 --- a/doc/unsorted/mailinglist.txt +++ b/doc/unsorted/mailinglist.txt @@ -6,9 +6,14 @@ Mailing List ============ -To subscribe to the mailing list for bcfg2 please visit: [https://lists.mcs.anl.gov/mailman/listinfo/bcfg-dev bcfg-dev mailman page] +To subscribe to the mailing list for bcfg2 please visit the `bcfg-dev +mailman page`_ -[http://dir.gmane.org/gmane.comp.sysutils.bcfg2.devel Searchable archives] are available from Gmane. You can also read the mailing list from any NNTP client via Gmane. +`Searchable archives`_ are available from Gmane. You can also read the +mailing list from any NNTP client via Gmane. + +.. _bcfg-dev mailman page: https://lists.mcs.anl.gov/mailman/listinfo/bcfg-dev +.. _Searchable archives: http://dir.gmane.org/gmane.comp.sysutils.bcfg2.devel .. raw:: html diff --git a/doc/unsorted/nisauth.txt b/doc/unsorted/nisauth.txt deleted file mode 100644 index 3dfad1d26..000000000 --- a/doc/unsorted/nisauth.txt +++ /dev/null @@ -1,19 +0,0 @@ -.. -*- mode: rst -*- - -.. _unsorted-nisauth: - -================== -NIS Authentication -================== - -'''Edit Django settings''' - -Django allows for custom authentication backends to its login procedure. Hostbase has an NIS authentication backend that verifies a user to be in the unix group allowed to modify Hostbase. - -To enable this feature: - -* first edit your {{{Hostbase/settings.py}}} file and uncomment the line {{{'Hostbase.backends.NISBackend',}}} in the list of {{{AUTHENTICATION_BACKENDS}}} -* enter the name of the unix group you want to give access to Hostbase in the {{{AUTHORIZED_GROUP}}} variable -* in your {{{Hostbase/hostbase/views.py}}} file at the very bottom, uncomment the block(s) of lines that give you the desired level of access - -Hostbase will now direct the user to a login page if he or she is not authorized to view a certain page. Users should log in with their regular Unix username and password. diff --git a/doc/unsorted/ssl.txt b/doc/unsorted/ssl.txt index 6189fdd3c..919f7ea71 100644 --- a/doc/unsorted/ssl.txt +++ b/doc/unsorted/ssl.txt @@ -8,9 +8,13 @@ Python SSL The ssl module can be found `here <http://pypi.python.org/pypi/ssl>`_. -With this change, SSL certificate based client authentication is supported. In order to use this, based CA-type capabilities are required. A central CA needs to be created, with each server and all clients getting a signed cert. See [wiki:Authentication] for details. +With this change, SSL certificate based client authentication is +supported. In order to use this, based CA-type capabilities are +required. A central CA needs to be created, with each server and all +clients getting a signed cert. See [wiki:Authentication] for details. -Setting up keys is accomplished with three settings, each in the "`[communication]`" section of bcfg2.conf:: +Setting up keys is accomplished with three settings, each in the +"`[communication]`" section of bcfg2.conf:: key = /path/to/ssl private key certificate = /path/to/signed cert for that key @@ -20,14 +24,23 @@ Setting up keys is accomplished with three settings, each in the "`[communicatio Python SSL Backport Packaging ============================= -Both the Bcfg2 server and client are able to use the in-tree ssl module included with python 2.6. The client is also able to still use M2Crypto. A python ssl backport exists for 2.3, 2.4, and 2.5. With this, M2Crypto is not needed, and tlslite is no longer included with bcfg2 sources. See [wiki:Authentication] for details. +Both the Bcfg2 server and client are able to use the in-tree ssl module +included with python 2.6. The client is also able to still use M2Crypto. A +python ssl backport exists for 2.3, 2.4, and 2.5. With this, M2Crypto +is not needed, and tlslite is no longer included with bcfg2 sources. See +[wiki:Authentication] for details. -To build a package of the ssl backport for .deb based distributions that don't ship with python 2.6, you can follow these instructions, which use [http://github.com/astraw/stdeb/tree/master stdeb]. Alternatively if you happen to have .deb packaging skills, it would be great to get policy-complaint .debs into the major deb-based distributions. +To build a package of the ssl backport for .deb based distributions +that don't ship with python 2.6, you can follow these instructions, +which use `stdeb`_. Alternatively if you happen to have .deb packaging +skills, it would be great to get policy-complaint .debs into the major +deb-based distributions. + +.. _stdeb: http://github.com/astraw/stdeb/tree/master The following commands were used to generate :download:`this -<python-ssl_1.14-1_amd64.deb>` debian package ('''NOTE:''' Version -numbers for the SSL module have changed). The `easy_install` command can -be found in the `python-setuptools` package.:: +<python-ssl_1.14-1_amd64.deb>` debian package The ``easy_install`` command +can be found in the `python-setuptools` package.:: sudo aptitude install python-all-dev fakeroot sudo easy_install stdeb @@ -39,6 +52,8 @@ be found in the `python-setuptools` package.:: dpkg-buildpackage -rfakeroot -uc -us sudo dpkg -i ../python-ssl_1.14-1_amd64.deb +.. note:: Version numbers for the SSL module have changed. + For complete bcfg2 goodness, you'll also want to package stdeb using stdeb. The completed debian package can be grabbed from :download:`here <python-stdeb_0.3-1_all.deb>`, which was generated using the following:: diff --git a/doc/unsorted/writing_specification.txt b/doc/unsorted/writing_specification.txt index 9fe847298..ce7630fde 100644 --- a/doc/unsorted/writing_specification.txt +++ b/doc/unsorted/writing_specification.txt @@ -13,7 +13,7 @@ Bcfg2 specifications are logically divided in to three areas: * Literal The metadata portion of the configuration assigns a client to its profile -group and to its non profle groups. The profile group is assigned +group and to its non-profile groups. The profile group is assigned in Metadata/clients.xml and the non profile group assignments are in Metadata/groups.xml. @@ -22,7 +22,7 @@ an abstract configuration for the client. An abstract configuration for a client identifies the configuration entities (packages, configuration files, service, etc) that a client requires, but it does not identify them explicitly. For instance an abstract configuration may identify -that a client needs the bcfg2 package with +that a client needs the Bcfg2 package with .. code-block:: xml @@ -353,11 +353,3 @@ Each of the generators is covered on their own page. || '' '''Plugin''' '' || '' '''Description''' '' || || [wiki:Plugins/Actions Actions] || Action entries are commands that are executed either before bundle installation, after bundle installation or both. || -|| [wiki:Plugins/Cfg Cfg] || The Cfg plugin provides a repository to describe configuration file contents for clients. In its simplest form, the Cfg repository is just a directory tree modeled off of the directory tree on your client machines. || -|| [wiki:Plugins/Deps Deps] || The deps plugin allows inter-entry dependencies (like package prerequisites) to be mapped independent of client roles. || -|| [wiki:Plugins/Hostbase Hostbase] || Hostbase is an IP management system built on top of Bcfg2. || -|| [wiki:Plugins/Pkgmgr Pkgmgr] || The Pkgmgr plugin define what packages are available for a host or image and where to find those packages. || -|| [wiki:Plugins/Rules Rules] || The Rules plugin is responsible for defining the details about the Directory, Permissions, !SymLink and Service elements. || -|| [wiki:Plugins/SSHbase SSHbase] || SSHbase is a purpose build bcfg2 plugin for managing ssh host keys. It is responsible for making ssh keys persist beyond a client rebuild and building a proper ssh_known_hosts file, including a correct localhost record for the current system. || -|| [wiki:Plugins/TCheetah TCheetah] || The TCheetah plugin allows you to use the [http://www.cheetahtemplate.org/ cheetah templating system] to create files, instead of the various diff-based methods offered by the Cfg plugin. It also allows you to include the results of probes executed on the client in the created files. || -|| [wiki:Plugins/TGenshi TGenshi] || The TGenshi plugin uses the [http://genshi.edgewall.org/ genshi toolkit] for generating files and functions much like the TCheetah plugin || |