wrote more detailed unit testing documentation

3 files changed, 372 insertions, 17 deletions

diff --git a/doc/conf.py b/doc/conf.py
index 538cd236d..4dda8327f 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -20,6 +20,7 @@ import time
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 sys.path.insert(0, os.path.abspath('../src/lib'))
+sys.path.insert(0, os.path.abspath('..'))
 
 # -- General configuration -----------------------------------------------------
 
@@ -263,7 +264,8 @@ def setup(app):
 versions = ["3.2", "2.7", "2.6"]
 cur_version = '.'.join(str(v) for v in sys.version_info[0:2])
 
-intersphinx_mapping = dict()
+intersphinx_mapping = dict(mock=('http://www.voidspace.org.uk/python/mock',
+                                 None))
 for pyver in versions:
     if pyver == cur_version:
         key = 'py'
diff --git a/doc/development/testing.txt b/doc/development/testing.txt
index 1f0842053..f00193574 100644
--- a/doc/development/testing.txt
+++ b/doc/development/testing.txt
@@ -8,22 +8,24 @@ Testing
 Testing Prereleases
 -------------------
 
-Before each release, several prereleases will be tagged. It is 
-helpful to have users test these releases (when feasible) because 
-it is hard to replicate the full range of potential reconfiguration 
-situations; between different operating systems, system management
-tools, and configuration specification variation, there can be 
-large differences between sites.
+Before each release, several prereleases will be tagged. It is helpful
+to have users test these releases (when feasible) because it is hard
+to replicate the full range of potential reconfiguration situations;
+between different operating systems, system management tools, and
+configuration specification variation, there can be large differences
+between sites.
 
 
-For more details please visit `Tracking Development Releases of Bcfg2 <http://trac.mcs.anl.gov/projects/bcfg2/wiki/TrackingDevelopmentTrunk>`_ .
+For more details please visit `Tracking Development Releases of Bcfg2
+<http://trac.mcs.anl.gov/projects/bcfg2/wiki/TrackingDevelopmentTrunk>`_
+.
 
 
 Upgrade Testing
 ---------------
 
-This section describes upgrade procedures to completely test the 
-client and server. These procedures can be used for either pre-release 
+This section describes upgrade procedures to completely test the
+client and server. These procedures can be used for either pre-release
 testing, or for confidence building in a new release.
 
 
@@ -33,9 +35,9 @@ Server Testing
 1. Ensure that the server produces the same configurations for clients
 
  * Before the upgrade, generate all client configurations using the
-   buildall subcommand of bcfg2-info. This subcommand takes a directory
-   argument; it will generate one client configuration in each file,
-   naming each according to the client name.
+   buildall subcommand of bcfg2-info. This subcommand takes a
+   directory argument; it will generate one client configuration in
+   each file, naming each according to the client name.
 
        .. code-block:: sh
 
@@ -54,7 +56,8 @@ Server Testing
  * Upgrade the server software
  * Generate all client configurations in a second location using the
    new software. Any tracebacks reflect bugs, and should be filed in
-   the ticketing system. Any new messages should be carefully examined.
+   the ticketing system. Any new messages should be carefully
+   examined.
  * Compare each file in the old directory to those in the new directory
    using ``bcfg2-admin compare -r /old/directory /new/directory``
 
@@ -78,5 +81,5 @@ Server Testing
 Client Testing
 ^^^^^^^^^^^^^^
 
-Run the client in dry-run and non-dry-run mode; ensure that multiple 
+Run the client in dry-run and non-dry-run mode; ensure that multiple
 runs produce consistent results.
diff --git a/doc/development/unit-testing.txt b/doc/development/unit-testing.txt
index 7af969686..a4ec2ad5c 100644
--- a/doc/development/unit-testing.txt
+++ b/doc/development/unit-testing.txt
@@ -6,12 +6,14 @@
 Bcfg2 unit testing
 ==================
 
-.. _Python Mock Module: http://python-mock.sourceforge.net/
+.. _Python Mock Module: http://www.voidspace.org.uk/python/mock
 .. _Python Nose: http://readthedocs.org/docs/nose/en/latest/
 
 You will first need to install the `Python Mock Module`_ and `Python
 Nose`_ modules. You can then run the existing tests with the
-following.::
+following:
+
+.. code-block: bash
 
     cd testsuite
     nosetests
@@ -26,3 +28,351 @@ You should see output something like the following::
 
 Unit tests are also run by Travis-CI, a free continuous integration
 service, at http://travis-ci.org/#!/Bcfg2/bcfg2/
+
+Testing in a virtualenv
+=======================
+
+Travis-CI runs the unit tests in a virtual environment, so to emulate
+that testing environment as closely as possible you can also use a
+virtual environment.  To do so, you must have `virtualenv
+<http://www.virtualenv.org/>`_ installed.
+
+There are two ways to test: Either with just the bare essential
+packages installed, or with optional packages installed as well.
+(Optional packages are things like Genshi; you can run Bcfg2 with them
+or without them.)  For completeness, the tests should be run in both
+manners.  (On Python 3, almost none of the optional packages are
+available, so it can only be run with just the required packages.)  To
+install the optional packages, set:
+
+.. code-block:: bash
+
+    export WITH_OPTIONAL_DEPS=yes
+
+This flag tells the install script to install optional dependencies as
+well as requirements.
+
+This assumes that you will create a virtual environment in
+``~/venvs/``, and that the Bcfg2 source tree is cloned into
+``~/bcfg2``.
+
+First, create a new virtual environment and activate it:
+
+.. code-block:: bash
+
+    cd ~/venvs
+    virtualenv travis
+    source travis/bin/activate
+
+Get the test suite from bcfg2:
+
+.. code-block:: bash
+
+    cp -R ~/bcfg2/* ~/venvs/travis/
+
+Next, you must install prerequisite packages that are required to
+build some of the required Python packages, and some optional packages
+that are much easier to install from binary (rather than from source).
+If you are running on Ubuntu (the platform Travis-CI runs on) and have
+sudo, you can simply run:
+
+.. code-block:: bash
+
+    testsuite/before_install.sh
+
+If not, you will need to examine ``testsuite/before_install.sh``
+and install the packages manually.  The equivalent for Fedora, for
+instance, would be:
+
+.. code-block:: bash
+
+    yum -y update
+    yum -y install swig pylint
+    if [[ "$WITH_OPTIONAL_DEPS" == "yes" ]]; then
+        sudo yum -y install libselinux-python pylibacl python-inotify \
+            PyYAML
+    fi
+
+Next, install required Python packages:
+
+.. code-block:: bash
+
+    testsuite/install.sh
+
+Install Bcfg2 itself to the virtualenv:
+
+.. code-block:: bash
+
+    pip install -e .
+
+Now you can run tests:
+
+.. code-block:: bash
+
+    nosetests testsuite
+
+Writing Unit Tests
+==================
+
+Bcfg2 makes extremely heavy use of object inheritance, which can make
+it challenging at times to write reusable tests.  For instance, when
+writing tests for the base :class:`Bcfg2.Server.Plugin.base.Plugin`
+class, which all Bcfg2 :ref:`server-plugins-index` inherit from via
+the :mod:`Plugin interfaces <Bcfg2.Server.Plugin.interfaces>`,
+yielding several levels of often-multiple inheritance.  To make this
+easier, our unit tests adhere to several design considerations:
+
+Inherit Tests
+-------------
+
+Our test objects should have inheritance trees that mirror the
+inheritance trees of their tested objects.  For instance, the
+:class:`Bcfg2.Server.Plugins.Metadata.Metadata` class definition is:
+
+.. code-block:: python
+
+    class Metadata(Bcfg2.Server.Plugin.Metadata,
+                   Bcfg2.Server.Plugin.Statistics,
+                   Bcfg2.Server.Plugin.DatabaseBacked):
+
+Consequently, the ``TestMetadata`` class definition is:
+
+.. code-block:: python
+
+    class TestMetadata(TestPlugin.TestMetadata,
+                       TestPlugin.TestStatistics,
+                       TestPlugin.TestDatabaseBacked):
+
+.. note::
+
+    The test object names are abbreviated because of the system of
+    relative imports in the ``testsuite`` tree, described below.
+
+This gives us a large number of tests basically "for free": all core
+:class:`Bcfg2.Server.Plugin.interfaces.Metadata`,
+:class:`Bcfg2.Server.Plugin.interfaces.Statistics`, and
+:class:`Bcfg2.Server.Plugin.helpers.DatabaseBacked` functionality is
+automatically tested on the ``Metadata`` class, which gives the test
+writer a lot of free functionality and also an easy list of which
+tests must be overridden to provide tests appropriate for the ``Metadata``
+class implementation.
+
+Additionally, a test class should have a class variable that describes
+the class that is being tested, and tests in that class should use
+that class variable to instantate the tested object.  For instance,
+the test for :class:`Bcfg2.Server.Plugin.helpers.DirectoryBacked`
+looks like this:
+
+.. code-block:: python
+
+    class TestDirectoryBacked(Bcfg2TestCase):
+        test_obj = DirectoryBacked
+        ...
+
+
+        def test_child_interface(self):
+            """ ensure that the child object has the correct interface """
+            self.assertTrue(hasattr(self.test_obj.__child__, "HandleEvent"))
+
+Then test objects that inherit from ``TestDirectoryBacked`` can
+override that object, and the ``test_child_interface`` test (e.g.)
+will still work.  For example:
+
+.. code-block:: python
+
+    class TestPropDirectoryBacked(TestDirectoryBacked):
+        test_obj = PropDirectoryBacked
+
+Finally, each test class must also provide a ``get_obj`` method that
+takes no required arguments and produces an instance of ``test_obj``.
+All test methods must use ``self.get_obj()`` to instantiate an object
+to be tested.
+
+An object that does not inherit from any other tested Bcfg2 objects
+should inherit from :class:`testsuite.common.Bcfg2TestCase`, described
+below.
+
+.. _development-unit-testing-relative-imports:
+
+Relative Imports
+----------------
+
+In order to reuse test code and allow for test inheritance, each test
+module should add all parent module paths to its ``sys.path``.  For
+instance, assuming a test in
+``testsuite/Testsrc/Testlib/TestServer/TestPlugins/TestMetadata.py``,
+the following paths should be added to ``sys.path``::
+
+    testsuite
+    testsuite/Testsrc
+    testsuite/Testsrc/Testlib
+    testsuite/Testsrc/Testlib/TestServer
+    testsuite/Testsrc/Testlib/TestServer/TestPlugins
+
+This must be done because Python 2.4, one of our target platforms,
+does not support relative imports.  An easy way to do this is to add
+the following snippet to the top of each test file:
+
+.. code-block:: python
+
+    import os
+    import sys
+
+    # add all parent testsuite directories to sys.path to allow (most)
+    # relative imports in python 2.4
+    path = os.path.dirname(__file__)
+    while path != "/":
+        if os.path.basename(path).lower().startswith("test"):
+            sys.path.append(path)
+        if os.path.basename(path) == "testsuite":
+            break
+        path = os.path.dirname(path)
+
+In addition, each new directory created in ``testsuite`` must contain
+an empty ``__init__.py``.
+
+This will allow you, within ``TestMetadata.py``, to import common test
+code and the parent objects the ``TestMetadata`` class will inherit from:
+
+.. code-block:: python
+
+    from common import inPy3k, call, builtins, u, can_skip, \
+        skip, skipIf, skipUnless, Bcfg2TestCase, DBModelTestCase, syncdb, \
+        patchIf, datastore
+    from TestPlugin import TestXMLFileBacked, TestMetadata as _TestMetadata, \
+        TestStatistics, TestDatabaseBacked
+
+Avoid Patching Where Possible
+-----------------------------
+
+The `Python Mock Module`_ provides a ``patch`` decorator that can be
+used to replace tested objects with ``Mock`` objects.  This is
+wonderful and necessary, but due to differences in the way various
+versions of Python and Python Mock handle object scope, it's not
+always reliable when combined with our system of test object
+inheritance.  Consequently, you should follow these rules when
+considering whether to use ``patch``:
+
+* If you need to mock an object that is not part of Bcfg2 (e.g., a
+  builtin or an object in another Python library), use ``patch``.
+* If you need to patch an object being tested in order to instantiate
+  it, use ``patch``, but see below.
+* If you need to patch a function (not a method) that is part of
+  Bcfg2, use ``patch``.
+* If you need to mock an object that is part of the object being
+  tested, do not use ``patch``.
+
+As an example of the last rule, assume you are writing tests for
+:class:`Bcfg2.Server.Plugin.helpers.FileBacked`.
+:func:`Bcfg2.Server.Plugin.helpers.FileBacked.HandleEvent` calls
+:func:`Bcfg2.Server.Plugin.helpers.FileBacked.Index`, so we need to
+mock the ``Index`` function.  This is the **wrong** way to do that:
+
+.. code-block:: python
+
+    class TestFileBacked(Bcfg2TestCase):
+        @patch("%s.open" % builtins)
+        @patch("Bcfg2.Server.Plugin.helpers.FileBacked.Index")
+        def test_HandleEvent(self, mock_Index, mock_open):
+            ...
+
+Tests that inherit from ``TestFileBacked`` will not reliably patch the
+correct ``Index`` function.  Instead, assign the object to be mocked
+directly:
+
+.. code-block:: python
+
+    class TestFileBacked(Bcfg2TestCase):
+        @patch("%s.open" % builtins)
+        def test_HandleEvent(self, mock_open):
+            fb = self.get_obj()
+            fb.Index = Mock()
+
+.. note::
+
+    ``@patch`` decorations are evaluated at compile-time, so a
+    workaround like this does **not** work:
+
+    .. code-block:: python
+
+        class TestFileBacked(Bcfg2TestCase):
+            @patch("%s.open" % builtins)
+            @patch("%s.%s.Index" % (self.test_obj.__module__,
+                                    self.test_obj.__name))
+            def test_HandleEvent(self, mock_Index, mock_open):
+                ...
+    
+    But see below about patching objects before instantiation.
+
+In some cases, you will need to patch an object in order to
+instantiate it.  For instance, consider
+:class:`Bcfg2.Server.Plugin.helpers.DirectoryBacked`, which attempts
+to set a file access monitor watch when it is instantiated.  This
+won't work during unit testing, so we have to patch
+:func:`Bcfg2.Server.Plugin.helpers.DirectoryBacked.add_directory_monitor`
+in order to successfully instantiate a ``DirectoryBacked`` object.  In
+order to do that, we need to patch the object being tested, which is a
+variable, but we need to evaluate the patch at run-time, not at
+compile time, in order to deal with inheritance.  This can be done
+with a ``@patch`` decorator on an inner function, e.g.:
+
+.. code-block:: python
+
+    class TestDirectoryBacked(Bcfg2TestCase):
+        test_obj = DirectoryBacked
+
+        def test__init(self):
+            @patch("%s.%s.add_directory_monitor" % (self.test_obj.__module__,
+                                                    self.test_obj.__name__))
+            def inner(mock_add_monitor):
+                db = self.test_obj(datastore, Mock())
+                mock_add_monitor.assert_called_with('')
+
+            inner()
+
+``inner()`` is patched when ``test__init()`` is called, and so
+``@patch()`` is called with the module and the name of the object
+being tested as defined by the test object (i.e., not as defined by
+the parent object).  If this is not done, then the patch will be
+applied at compile-time and ``add_directory_monitor`` will be patched
+on the ``DirectoryBacked`` class instead of on the class to be tested.
+
+Some of our older unit tests do not follow these rules religiously, so
+as more tests are written that inherit from larger portions of the
+``testsuite`` tree they may need to be refactored.
+
+Naming
+------
+
+In order to make the system of inheritance we implement possible, we
+must follow these naming conventions fairly religiously.
+
+* Test classes are given the name of the object to be tested with
+  ``Test`` prepended.  E.g., the test for the
+  :class:`Bcfg2.Server.Plugins.Metadata.Metadata` is named
+  ``TestMetadata``.
+* Test classes that test miscellaneous functions in a module are named
+  ``TestFunctions``.
+* Test modules are given the name of the module to be tested with
+  ``Test`` prepended.  Tests for ``__init__.py`` are named
+  ``Test_init.py`` (one underscore).
+* Tests for methods or functions are given the name of the method or
+  function to be tested with ``test_`` prepended.  E.g., the test for
+  :class:`Bcfg2.Server.Plugin.helpers.StructFile.Match` is called
+  ``test_Match``; the test for
+  :class:`Bcfg2.Server.Plugin.helpers.StructFile._match` is called
+  ``test__match``.
+* Tests for magic methods -- those that start and end with double
+  underscores -- are named ``test__<name>``, where name is the name of
+  the magic method without underscores.  E.g., a test for ``__init__``
+  is called ``test__init``, and a test for ``__getitem__`` is called
+  ``test__getitem``. If this causes a collision with a non-magic
+  function (e.g., if a class also has a function called
+  ``_getitem()``, the test for which would also be called
+  ``test__getitem``, seriously consider refactoring the code for the
+  class.
+
+Common Test Code
+----------------
+
+.. automodule:: testsuite.common