Extend VHC
**********

VHC provides a few extension hooks to allow to add custom drivers without need
to edit the VHC code.

We provide these three access points:

* :ref:`cdriver`
* :ref:`cparser`
* :ref:`hooks`

For the sake of this documentation we assume that the custom code is
implemented into a package with the following structure:

.. code-block:: bash

    vhc_extra
    ├── setup.py                 # make the package installable
    ├── vhc_extra                # the actual code
    │   ├── __init__.py          # this is a package
    │   ├── common.py            # module implementing the driver
    │   ├── ref_file_parsers.py  # module implementing new file parsers
    │   └── hooks.py             # vhc modifications
    ├── ...   
    ...

.. _whereis:

Where is the code?
==================

Before we start, we need to let VHC know where the extra code, ``vhc_extra`` is.

We recommend to install the package, so that ends up in the standard path known
by Python.

Alternatively you can do one of the following two:

* put the path to your package into the ``PYTHONPATH`` before running ``vhc``

    .. code-block:: bash

        export PYTHONPATH=/path/to/vhc_extra:$PYTHONPATH

* add the path into the ``vhc_settings.cfg`` file

    .. code-block:: ini

        [extension]
        # list of comma separated paths to add to the python path
        extra_paths = /path/to/vhc_extra
        
``vhc`` will add the extra paths, unless it's already present, just after
setting up the configuration, the logging objects and, if required, the
multiprocessing capability.

We strongly suggest against adding ``vhc_extra/vhc_extra`` to the
``PYTHONPATH`` in order to avoid name clashes, e.g. between the ``common``
module in ``vhc_extra`` and :mod:`libvhc.common`

New recipes
===========

If you implement a new recipe, add it to the list of known recipes in the
configuration file:

.. code-block:: ini

    [recipes]
    # comma separated list of recipes and of paths where the drivers are
    # implemented. The `recipes_paths` are prepended to the `libvhc` and the system
    # packages.
    recipes = hetdex_dithers, flat, arc, bias, new_recipe

When it read the recipe file, ``vhc`` compare it with this list and abort if the
recipe is not known.

If you add a new recipe and use one of the factories described in
:doc:`custom_drivers`, you must add a section with at least the expected number
of exposures; e.g.:

.. code-block:: ini

    [new_recipe]
    # expected number of exposures
    n_exposures = 3


.. _cdriver:

Plug the custom driver
======================

Let's assume that in file ``vhc_extra/common.py`` you implement a driver
``do_something``. If you cannot or do not want to use any of the factories
described in :doc:`custom_drivers`, remember that driver must have the
signature shown by :func:`libvhc.function_signature`; if this is not the case,
the driver will fail. ``vhc`` will continue running but the failure will be
notified in the ``v_results.txt``, ``log.txt`` files and in the html recap.

To use the driver simply add the following line to the relevant ``v_driver.txt`` files::

    vhc_extra.common:do_something

Once ``vhc`` will hit this line in the driver file, it will it will import the
``vhc_extra.common`` module get the ``do_something`` function and execute it.

If the module name is very long and/or you use various drivers from the
package, you can shorten the driver name. If you have, let's say, two drivers
``do_something``, ``do_something_else`` in the module
``my_very_long_package.also_very_long.driver`` you can shorten the driver names
in the ``v_driver.txt`` files to::

    driver:do_something
    driver:do_something_else

but you have to tell ``vhc`` where to look for them. You can do it adding the
first part of the package name to the ``vhc_settings.cfg`` file:

.. code-block:: ini

    [extension]
    # list of comma separated python modules parents of the extra drivers
    driver_parent_modules = my_very_long_package.also_very_long

A note on driver loading mechanism
----------------------------------

At this point you ought to know how drivers are loaded and executed.

#. ``vhc`` looks for a ``v_driver.txt`` file in the input path: if it finds it,
   it reads all its content. Otherwise the files is created and filled with a
   default set of drivers for the given recipe (see :mod:`libvhc.utils`).
#. :mod:`libvhc` and all the modules provided by the configuration variable
   ``driver_parent_modules`` in the ``extension`` section are imported.
#. ``vhc`` cycles through the list of drivers. For each of them:

    * split the module from the function name.
    * it tries to import the module first as absolute, then as submodule of
      ``libvhc`` and of the ``driver_parent_modules``; for any of the
      successful imports, it tries to get the function. If it can't load the
      module at all, or can't find the function in any of the loaded modules
      the driver is logged as a critical failure
    * if it can find the function, ``vhc`` executes it: if this fails the
      driver is again logged as a critical failure.

.. _cparser:

Custom reference file parsers
=============================

What if your new driver needs to parse a reference file just like the one
described in :mod:`libvhc.reference_file_parser` and you want to reuse all the
machinery already implemented?

In your ``ref_file_parsers.py`` file subclass
:class:`libvhc.reference_file_parser._BaseParser`, or a derived one, and
override the :meth:`~libvhc.reference_file_parser._BaseParser.filename`. If you
want it to be registered by ``vhc`` and accessible via
:func:`~libvhc.reference_file_parser.picker` just do so. First mark the new
parser for registration:

.. code-block:: python

    import libvhc.reference_file_parser as vhcrfp

    @vhcrfp.register('parse_something')
    class MyParser(vhcrfp._BaseParser):
        def filename(self):
            """returns the file name from the configuration"""
            return self.conf.get("extra_vhc", "some_reference")

        def parse_value(self, str_value):
            """parse the extra columns at need"""
            return str_value.split()
            
Then add the module name to the VHC configuration file.

.. code-block:: ini

    [extension]
    # list of comma separated python modules implementing extra reference file
    # parsers. E.g.:
    # mypackage.mymodule
    reference_parser_extra = vhc_extra.ref_file_parsers

If you have done this you can get the initialised parser in your driver
implementation calling ``vhcrfp.picker('parse_something')`` and obtain the
relevant information with the method
:meth:`~libvhc.reference_file_parser._BaseParser.get_value`.

.. _hooks:

Custom hooks
============

Execute functions
-----------------

There is a third, deeper way of customise VHC: the possibility to call hooks
early in ``vhc``.

Let's suppose that you want to:

#. load an external configuration file: e.g. we need to know where the reference
   file is located;
#. add a new file name match pattern for your new recipe: we can use the
   factories to find the files and, if does not already exists, we can
   automatically create the recipe file of the correct type;
#. extend the list of default drivers: so if the driver file does not exists we
   can use a default set for the new recipe

You can get it writing three functions in ``vhc_extra/hooks.py`` and tell
``vhc`` where to find them using the configuration file

.. code-block:: ini

    [extension]

    # list of comma separated modules and, optionally, callables, to be loaded and
    # called after the configuration and logger objects have been setup. E.g.:
    # mypackage.mymodule
    # mypackage.mymodule:function, mypackage.mymodule2:func2
    hooks = vhc_extra.hooks:load_conf, vhc_extra.hooks:new_recipe_pattern, vhc_extra.hooks:def_drivers 

When loading and executing the hook functions,
:func:`libvhc.loaders.execute_hooks`, ``vhc`` passes to them the configuration
instance name and the logger instances.

Here is an example of the three functions:

.. code-block:: python

    import libvhc.config as vhcconf
    import libvhc.utils as vhcutils

    def load_conf(conf_name, logger):
        """load an extra configuration file
        """
        extra_conf_file = "/path/to/conf.cfg"
        conf = vhcconf.get_config(name=conf_name)
        conf.read(extra_conf_file)
        # put back the updated configuration instance
        vhcconf._config_dic[conf_name] = conf 


    def new_recipe_pattern(*_):
        """insert a new recipe pattern match into vhc
        
        We don't care about any of the input parameters

        The new file name looks like:: 

              20120301T000838_045LL_nrec.fits
        """
        vhcutils.recipe_match['new_recipe'] = "*[0-9]*nrec.fits",
        
    def def_drivers(conf_name, logger):
        """Create default drivers for our new recipe
        """
        new_drivers = vhcutils.common_drivers
        new_drivers += ['vhc_extra.common:do_something']
        vhcutils.default_drivers['new_recipe'] = new_drivers
        logger.info("new default drivers added")

Of course you can create an fourth function that calls them

.. code-block:: python

    def init(conf_name, logger):
        load_conf(conf_name, logger)
        new_recipe_pattern()
        default_drivers(conf_name, logger)

and tell ``vhc`` about it

.. code-block:: ini

    [extension]
    hooks = vhc_extra.hooks:init

Load modules
------------

If your initialisation can all be done at the module level and doesn't need any
input parameter, you can also skip passing the function name. E.g.:

.. code-block:: python

    def new_recipe_pattern():
        """insert a new recipe pattern match into vhc
        
        We don't care about any of the input parameters

        The new file name looks like:: 

              20120301T000838_045LL_nrec.fits
        """
        vhcutils.recipe_match['new_recipe'] = "*[0-9]*nrec.fits",
        
    def def_drivers():
        """Create default drivers for our new recipe
        """
        new_drivers = vhcutils.common_drivers
        new_drivers += ['vhc_extra.common:do_something']
        vhcutils.default_drivers['new_recipe'] = new_drivers

   new_recipe_pattern()
   def_drivers()

.. code-block:: ini

    [extension]
    hooks = vhc_extra.hooks

This will load the module ``vhc_extra.hooks`` and, as side effect, execute the
two functions.

Bonus
=====

If you want to do more you can, probably.

The ``vhc`` executable simply calls :func:`libvhc.vhc.main`. If you want to run
some code before starting ``vhc``, you can write a function that does what you
need and call :func:`~libvhc.vhc.main`. E.g. in a ``entry_point.py`` module
you can define:

.. code-block:: python

    import libvhc.vhc as vhc

    def main(argv=None):

        # do the setup that you need
        [...]
        # call vhc
        vhc.main(argv=argv)
        # do any cleanup if needed
        [...]

Then add the entry point in your ``setup.py`` file to create a new ``vhc_extra``
executable when installing your package

.. code-block:: python

    entry_points={'console_scripts': ['vhc_extra = vhc_extra.entry_point:main']}

Enjoy your customisations!