Board Directive:

• At the behest of the Board, HET will start full time science without any routine engineering periods
• HET may request engineering time with the HET Scientist before the call for proposals for a trimester if engineering tests will preclude science. Once granted this engineering time may be scheduled by the operations team as needed.

Goals

• install small, incremental changes without interrupting science or causing long down times.
• deploy larger changes in a controlled fashion
• be able to switch back to a working version of software if a change does not work.
  • this allows use to do short engineering runs of 1-2 hours and then revert back to working code for science
• test changes as completely as possible during the day time
• keep track of which version is running at which time
• report to operations team the changes and impact of the changes to be installed.

Development

You can work in your own branch or trunk. For mid-sized or extensive modifications that may require intermediate commits, a branch is recommended. Once you have tested as fully as possible the branch can be deployed in order to test on the hardware during the day or on the sky at night. If your changes don't work, redeploy the previous tag (see Deployment below). Once you are confident that your modification works, then do a single merge with trunk. In this manner, only one revision needs to be merged or back out in order to add or remove your modification. When working in a branch, you should merge trunk frequently to pick up the current working version

If you are making only small changes that need only one or two commits, you can work in trunk... but don't break the build.

Testing is important prior to releasing code on unsuspecting RA/TOs. With a new deployment scheme it is easy to revert to a prior working edition.

If you need to test only your modification at night, then create a branch, make your changes, commit, and then install only the package(s) with the changes using the '-p' flag from het_release. Allow your version to run at night but be prepared to reinstall the current build (by running make install [packages] in the current build directory).

If testing during daytime only, then you can run the packages that have the changes. Replace the existing code before night operations are reinstall the existing version by doing make install [packages] in the current build directory. In this way we can keep track of what version are running at night during science operations.

Deployment

• Old Deployment method
  • roughly once per month create a new branch from trunk with the name branches/deployment/science/YYYYMMDD
  • build and install the branch
  • merge important bug fixes into the branch as necessary

• New Deployment method (still being debugged)
  • is done via the script het_release, for example I would use the following
    • het_release -U fowler -a jrf@utexas.edu -b /opt/het/hetdex/src/tcs -r het-operations@het.as.utexas.edu -i /opt/het/hetdex -u hetdex:hetdex -P trunk
    • the command line flags for het_release are shown below. Note that -i is required for installation and -b is highly recommended. I use -U because my user name on hetdex-pr is different that my user name on the machines at Het. The -P flag defaults to trunk but must be used if you want to install a branch or a tag.

$ ./het_release -h
Usage: ./het_release [-b <build dir>] [-c <changelog>] [-d] [-h] [-i <install dir>] [-l <log file>]
           [-p <packages>] [-P <repo path>] [-r <recipients>] [-R <repo>] [-t <tag>]
           [-u <user>[:<group>]] [-U <repo user>]

Tags the HET <source repository path> at <tag> then checks the tag out, builds and
optionally installs the HET software <packages>.

NOTE: This script likely doesn't support file paths with special characters and/or spaces.

Options are:

  -a <email author> Sets the author for the email notification for the build.
  -b <build dir>    Specifies the path where the <tag> will be checked out and built.
                    Defaults to /tmp.
  -c <changelog>    Copies the <changelog> to the given file.  The temporary file used
                    for the changelog is removed without copying by default.
  -d                Enable debug output. To disable system and svn commands, give '-dd'.
  -h                Print this message and exit.
  -i <install dir>  Specifies the directory where the build will be installed.  If omitted
                    or given an empty string, the installation is skipped.  Defaults to
                    "".
  -l <log path>     Writes the script console log and build log under <log path>/.  If empty,
                    the log is written to the current working directory.  Defaults to /tmp.
  -p <packages>     List of systems to build.  Packages supported are determined by top
                    level directories and makefile targets.  Defaults to all.
  -P <repo path>    Path of the installation source in the repository.  Defaults to trunk.
  -r <addresses>    Specify a comma separated list of email <addresses> that will
                    be sent a copy of the build and changelog.
  -R <repo>         Remote location of the repository.
                    Defaults to svn+ssh://hetdex-pr.as.utexas.edu/repos/het.
  -t <tag>          Specify the repository path for the tag.  This is the relative
                    path from the root given by <repo>.
                    Defaults to tags/deployment/liberty_20180904T145132.
  -u <user>:<group> Sets ownership of the insallation to <user>:<group>.
                    Defaults to leaving original installation permissions set.
  -U <user>         Username for all svn transactions, defaults to the current user.

• you may deploy/install trunk or any previous tag or branch
• the het_release script does the following steps,
  • start-up administrative work, e.g. setup logs, initialize variables
  • record all the variable settings in the output file
  • tag the current deployment as tags/deployment/host_name_YYYYMMDDThhmmss
  • check out a clean copy of the tag into /tmp or the directory specified by the -b command line flag
  • build (Note that standard makefile.local files are pre-built for htcs, vdas, lrs2, etc.
  • install the executable programs and scripts in the deployment directory, given by the -i command line flag
  • record results in the output file
  • email a report based on the output log the the email address specified by the -r command line flag
• the servers will need to be restarted manually
• you can generate a list of changes between a working copy and the trunk with
  • cd /opt/het/hetdex/src/tcs/build
  • generate-changelog svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/trunk .

Decisions Required

• decide whether or not to clean out the directory before installing new files
• decide whether to split the build/installation of the camra code and the scripts from main tcs servers
  • if so, create deployment directories for each, e.g. /opt/het/tcs, /opt/het/camra with ./bin, ./etc, ./lib as required
  • decide if we build camra on vdas/lrs separate from a build on htcs, build/install script can handle this requirement
• should third-party libraries be common across hosts, e.g. ZMQ, Qt, gsl
• does this methodology support a clean migration path to later versions or the libraries as well as to Redhat 7?
• write and test het_release script (mostly complete)

Previous notes from Jason (which this procedure is partially based on)

• Any deployment, science or otherwise, will be done from a fresh checkout of the repository.
• The version of repos/het installed will be tagged, repos/het/tags/deployment/YYYYmmddTHHMMs
• The target installation directories for repos/het should signify the source, this allows those directories to be refreshed in bulk rather than piecewise. Utilities that live outside of repos/het will not reside in the repos/het installation path. The install process will delete the prior installation, so that we don't run scripts from one version against tcs at another. (ok, we can just issue a warning during the install if we find something unexpected)
• An email will be sent to the group noting the change from one version to another, the details of the related repository pointers, the changelog, and any errors/warnings emitted by the build.
• How to segregate camra configs from this process?
• The script will be named "release", for now
• Usage is: release /path/to/repo, i.e. release ​svn+ssh://hetdex-pr/repos/het/branches/my_fun_test_branch
• Deployments will be stored in a common path
• The relevant email will contain the source locations of the before and after checkouts
• Deployment emails from instrument subsystems will be easily disambiguated from emails related to the general tcs
• Support scripting the install across multiple architectures with a single command?
• Checkouts related to non-before-or-after installs will be automatically deleted, assuming no local modifications. Otherwise a note about the prior-prior will be included in the relevant email, noting why it couldn't be removed.
• Only two copies per architecture (at most three, see above) will be on disk at HET, aside from developer checkouts. Only these 3 revisions will be referenced, at any point in time, in tcs communiques.
• Everything will go through a repository, whatever the technology.
• The process of rebuilding will include restarting the relevant services.
• Remove all files before installing new ones? Will have to move non-tcs executables and libraries to another location
• split camra, configs, tcs, hetlib?
• better issues and change tracking
• build/deploy for multiple architectures, RHEL 6/7, Het/107", use autoconf if we need to build on different architectures?