Notes about building and running the WFU software.
J. R. Fowler
15 Nov 2013
23 Apr 2014 moved code from liberty to htcs

Requirements
  g++ compiler currently using g++-4.4.7 as distributed by Redhat
  gmake currently using 3.81 as distributed by Redhat

Additional software required

  To install the WFU and Hetdex software we need a few auxillary
packages that are not normally installed under Redhat Linux.
Currently I am installing these packages in;

  export INSTALLDIR=/opt/het/hetdex.
  (sudo not required as of 30 Nov 2013, all directories own by jrf now)

  zeromq from http://zeromq.org, a socket and communications library.
   (hetdex-pr uses version 3.2.2 but the latest stable version from the web site is 4.0.3.)
   (install 4.0.3)
  ./configure --prefix=$INSTALLDIR
  make
  make install
  zeromq 4.3.4   24 May 2021
  https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.tar.gz
   ./configure --prefix=$INSTALLDIR
   make
   make install


  czmq from http://czmq.zeromq.org provides high level C-bindings for zeromq.
   (hetdex-pr uses version 1.3.2 while the latest stable version at the web site is 2.0.3)
   (install 2.0.3)
   ./configure --prefix=$INSTALLDIR --with-libzmq=$INSTALLDIR
   make
   make install
  czmq 3.0.2
   ./configure --prefix=$INSTALLDIR --with-libzmq=$INSTALLDIR --with-makecert=yes
   make
   make install
  czmq 4.12.1  24 May 2021
  https://github.com/zeromq/czmq/releases/download/v4.2.1/czmq-4.2.1.tar.gz
   ./configure --prefix=$INSTALLDIR --with-libzmq=$INSTALLDIR
   make
   make install

  cfitsio from http://heasarc.gsfc.nasa.gov/fitsio/
   (hetdex-pr uses version 3.30, lastest release is 3.36.)
   (install 3.36)
   ./configure --prefix=$INSTALLDIR
   make
   make install

  old ltl instructions
  ltl (Little Template Library) from hetdex-pr:/home/hetdex/ltl-1.9.10.tar.gz
    The above file failed to link properly with PAS software
    Installed /home/het_auxil/ltl.niv instead
  ./configure --prefix=$INSTALLDIR
  make   (2 warnings about using largefile mode!)
  sudo make install

   23 April 2014 Installed from ltl.tgz, a Niv version from 2014-03-14 which includes only
   the includes files and the libraries but not the source.
  cd ltl
  sudo cp -r include/ltl $INSTALLDIR/include/ltl
  sudo cp lib/libtlt.a $INSTALLDIR/lib

  25 Aug 2014 New ltl instructions
    get ltl from the repository
    svn co svn://luna.mpe.mpg.de/ltl/trunk
    ./bootstrap
    # ./configure --prefix=$INSTALLDIR CXXFLAGS='-O2'
    ./configure <normal options> CXXFLAGS="-fPIC" CFLAGS="-fPIC" # 2015 instruction from Jason
    make
    make install


  slalib astrometry library by Pat Wallace from hetdex-pr:/home/het_auxil/slalib.tgz
  edit Slalib/Makefile
    change
     SLA_LIB_DIR=/opt/hetdex/lib/

     SLA_INC_DIR=/opt/hetdex/include
  make (this will try to install but will have a permission error)
  sudo make

     The 'make' command compiles all the objects files under the 'jrf'
     ownership but 'make' also tries to install the library so the
     compile will have an error.  By doing the 'sudo make' afterwards
     the libraries are installed with root ownership but the object
     file remain owned by 'jrf'

  chmod 644 /opt/hetdex/include/sla*

     The *.h files are installed with execute permission, this removes it.

  Edit slamac.h. Place #ifndef around the definitions of TRUE and FALSE
  in order to avoid conflicts with other packages.

  ln -s /opt/hetdex/lib/libSlalib.a /opt/hetdex/lib/libslalib.a


  sqlite from http://www.sqlite.org
    (hetdex-pr uses version 3.7.17, latest available is 3.8.2)
    (install 3.8.2)
  ./configure --prefix=$INSTALLDIR
  make
  make install


  epics from http://www.aps.anl.gov/epics/
   (hetdex-pr uses version 3.17.7, released July 2002), latest release 3.14.12.3 released 2012-12-17)
   export EPICS_BASE=/opt/hetdex/src/base-3.14.12.3
   export EPICS_HOST_ARCH=`$EPICS_BASE/startup/EpicsHostArch.pl`

   Modify $EPICS_BASE/configure/CONFIG_SITE:INSTALL_LOCATION to change installation directory.
    (installed in /opt/hetdex/epics  as a test, currently owned by jrf. Use sudo make if owned by root)

   make clean
   make  # This installs bin, configure, db, dbd, html, include, lib, and templets
         # in INSTALL_LOCATION if defined or in $EPICS_BASE by default


  PvApi this is commercial software from ProSilica
   copied hetdex-pr:/home/het_auxil/PvApi to /opt/hetdex/PvApi


  QT 4.8.5 from http://qt-projects.org
    ./configure --prefix=$INSTALLDIR  # this creates lots of output
       open source  # accept open source code
       yes          # accept license
    make 
    make install


Install additional python packages 

  Sphinx 1.2.3 from http://sphinx-doc.org
    python setup.py build
    python setup.py install --prefix=$INSTALLDIR
    make test

  nose 1.3.7 from https://pypi.python.org/pypi/nose
  (for local Python 2.7 only,nose 1.3.6 installed with python 2.6)
    python setup.py install --prefix=$INSTALLDIR

  numpy 1.9.1 from http://scipy.org
    #sudo python setup.py config_fc --fcompiler=gnu95  install --prefix=$INSTALLDIR
    #cd
  numpy 1.12.1 from http://scipy.org   2017/04/11
    pip install numpy --prefix $INSTALLDIR
    python -c 'import numpy; numpy.test()'

  scipy 0.14.0 fron http://scipy.org
    #python setup.py install --prefix=$INSTALLDIR
  scipy 0.19.0 fron http://scipy.org   2017/04/11
    pip install scipy --prefix $INSTALLDIR

  mock 1.0.1 from https://pypi.python.org/simple/mock/
  (for matplotlib installation under python 2.7)
    python setup.py install --prefix=$INSTALLDIR

  matplotlib 1.4.2 from http://matplotlib.org
    python setup.py install --prefix=$INSTALLDIR

    if there you get the error "ImportError: No module named functools_lru_cache", remove
    the backports from /opt/het/hetdex/lib/python2.7/site-packages with
      pip uninstall backports.functools_lru_cache
    and reinstall it in the Python site-packages directory
      pip install backports.functools_lru_cache  # Note there is no --prefix flag

  astropy 0.4.2 from http://www.astropy.org
  astropy 1.0.4 from https://pypi.python.org/astropy
    python setup.py --offline install --prefix=$INSTALLDIR

  numpydoc 0.5 from https://pypi.python.org/pypi/numpydoc
    python setup.py install --prefix=$INSTALLDIR

  python 2.7.9 from https.//www.python.orig/downloads/release/python-279
    ./configure --enable-shared --prefix=$INSTALLDIR/Python-2.7;
    make;
    make install;

  pil python image library from http://pythonware.com/products/pil/
    python setup.py build_ext -i
    python selftest.py
    python setup.py install --prefix=$INSTALLDIR

  pyzmq 14.7.0 from https://pypi.python.org/pypi/pyzmq
    python setup.py install --prefix=$INSTALLDIR

  sphinx-extensions from https://bitbucket.org/birkenfeld/sphinx-contrib/
    (needed for pyzmq documentation)
    cd napoleon
    python setup.py install --prefix=$INSTALLDIR

  Sip 4.16.0 from http://www.riverbankcomputing.com/software/sip/download
    (required for PyQt)
    python configure.py
    make
    make install

  PyQt 4.11.4 from http://www.riverbankcomputing.co.uk/software/pyqt/download
   python configure-ng.py
   make
   make install

  Pyraf 2.1.10 http://www.stsci.edu/institute/software_hardware/pyraf/current/download
    cd ./required_pkgs/d2to1
    python setup.py install --prefix=$INSTALLDIR
    cd ../stsci.tools
    python setup.py install --prefix=$INSTALLDIR
    cd ../..
    python setup.py install --prefix=$INSTALLDIR

  Pip 8.1.1 https://pypi.python.org/pypi/pip
    python setup.py install --prefix=$INSTALLDIR
    update to 9.0.1 with command
    pip install --upgrade --prefix=/opt/het/hetdex pip

  Dateutil 2.5.3 https://pypi.python.org/pypi/python-dateutil/2.5.3
    cd ./python-dateutil-2.5.3
    python setup.py install --prefix=$INSTALLDIR

  Pandas
    pip install --prefix=/opt/het/hetdex pandas
    pip install --upgrade --prefix=/opt/het/hetdex/ pytz

  Gyp build system
    git clone https://chromium.googlesource.com/external/gyp
    python setup.py install --prefix=$INSTALLDIR

  PyEphem 3.7.6.0 http://rhodesmill.org/pyephem
    pip install pyephem --prefix=$INSTALLDIR


   For the Trac wiki
   on hetrac use python2.6 as that is what httpd wants

  Genshi 0.7 from http://genshi.edgewall.org/wiki/Download
    sudo python setup.py install
    Install documentation (HTML files already exist)
      cd doc
      cp -R * /opt/het/hetdex/doc/Genshi
      (./Genshi/index.html)

  Trac 1.0.9 from http://trac.edgewall.org/wiki/TracDownload
    sudo python setup.py install
    mkdir /data1/het/www/html/trac/WFUCommissioning
    trac-admin /data1/het/www/html/trac/WFUCommissioning initenv
    sudo chown -R apache.apache /data1/het/www/html/trac/WFUCommissioning
    trac-admin /usr/share/trac/projects/my-project deploy /tmp/deploy
    mv /tmp/deploy/* /usr/share/trac
    #modify /etc/httpd/conf/httpd.conf 
    # add users on hettrac
    sudo htdigest /var/www/html/trac/trac.htpasswd trac <username>
    sudo trac-admin /var/www/html/trac/WFUCommissioning persmission add <username> WIKI_ADMIN

   For mailing lists
  dnspython 1.12.0 from http://www.dnspython.org/
    python setup.py install --prefix=$INSTALLDIR

  Mailman 2.1.20 from https://www.gnu.org/software/mailman/
    # on poster and htcs
    sudo groupadd mailman
    sudo useradd -c"GNU Mailman" -s /no/shell -d /no/home -g mailman mailman
    # there will be error messages about creating /no/home, ignore them
    cd /opt/het/hetdex
    mkdir mailman
    sudo chgrp mailman mailman
    sudo chmod a+rx,g+ws mailman
    cd /opt/het/hetdex/src/mailman-2.1.20
    ./configure --with-mailhost=poster.as.utexas.edu --prefix=/opt/het/hetdex/mailman --with-cgi-gid=apache --with-mail-gid=mail
    make
    make install
    cd /opt/het/hetdex/mailman
    sudo ./bin/check_perms -f
    # on poster
    cd /etc/smrsh
    ln -s /opt/het/hetdex/mailman/mail/mailman mailman
    cd /opt/het/hetdex/mailman/
    ./bin/newlist mailman
    # edit /etc/aliases
    sudo newaliases
    cd /opt/het/hetdex/mailman/cron
    sudo crontab -u mailman crontab.in
    sudo crontab -e -u mailman
    # change to UT time.
    edit Mailman/mm-cfg.py
    run ./bin/withlist -l -r fix_url <list-name>


Virus software

  PyHetdex
    (required for hetdex shuffle)
    svn checkout svn://luna.mpe.mpg.de/pyhetdex/trunk pyhetdex
    python setup.py install --prefix=$INSTALLDIR
    (installed Pillow 2.9.0 as well, may have overwritten the PIL installation)
    some issues with astropy and matplotlib
    # new method 8 April 2016
    pip install --prefix /opt/het/hetdex --extra-index-url https://gate.mpe.mpg.de/pypi/simple/ pyhetdex


  pyds9 1.8 from https://github.com/ericmandel/pyds9
    (required by hetdex shuffle)
    unzip Orig/pyds9-master.zip
    cd pyds9-master
    python setup.py install --prefix=$INSTALLDIR

  Hetdex Shuffle
   svn co svn://luna.mpe.mpg.de/hetdexshuffle/trunk hetdexshuffle
   follow directions in README.txt

  Alabaster 0.7.6 from https://pypi.python.org/pypi/alabaster#downloads
    (required for pyhetdex documentation)
    python setup.py install --prefix=$INSTALLDIR

  VHC_config
   #!Currently installed from /home/mcs/astronomer/hetdex_software
     svn co svn://luna.mpe.mpg.de/vhc_config/trunk vhc_config
     mkdir $INSTALLDIR/etc/Virus
     cd $INSTALLDIR/etc/Virus
     ln -s $INSTALLDIR/src/vnc_config
     # update
     cd Virus/vhc_config
     svn up

  Cure
    #!The current version of cure is in /home/mcs/astronomer/hetdex_code/cure
    #!No need to change the Makefiles as this is a local installation
    # Can now build and install in /opt/het/hetdex 18 April 2018

    svn checkout svn://luna.mpe.mpg.de/cure/trunk cure
    cd cure                                           
    Change Makefile
     change HAVE_OPENGL to no
     change
       LTLI=../../ltl/
       LTLL=../../ltl/ltl
     to
       LTLI=/opt/het/hetdex/include
       LTLL=/opt/het/hetdex/lib
     and
       BINDIR=$(CURRENT_DIRECTORY)/bin
     to
       BINDIR=/opt/het/hetdex/bin
     and
	cp scripts/parallel bin/
     to
	cp scripts/parallel $(BINDIR)
     and
        LAPACK_LIB=-L/usr/lib64/atlas -lf77blas -llapack -latlas
     to
        LAPACK_LIB=-L/usr/lib64/atlas -llapack -lcblas -latlas

     Change pasphot/Makefile
       LTLI=../../ltl
     to
       LTLI=/opt/het/hetdex/include

    make
    make install

    # To update cure
    cd cure
    svn up
    make
    make install

  Vhc
   #!Currently installed from /home/mcs/astronomer/hetdex_software
    svn checkout svn://luna.mpe.mpg.de/vhc/trunk vhc
    python setup.py install --prefix=$INSTALLDIR
    # use instead of setup.py Sept 2016
    pip install --prefix /opt/het/hetdex --extra-index-url https://gate.mpe.mpg.de/pypi/simple/ vhc
    # to update
    pip install --upgrade --prefix /opt/het/hetdex --extra-index-url https://gate.mpe.mpg.de/pypi/simple/ vhc


  Vdat
    (requires libyaml)
    svn checkout svn://luna.mpe.mpg.de/vdat/trunk vdat
    python setup.py install --prefix=$INSTALLDIR
    # new method 8 April 2015
    pip install --prefix /opt/het/hetdex --extra-index-url https://gate.mpe.mpg.de/pypi/simple/ vdat

  OCD
    installed on /opt/het/hetdex
    # Not installed on pypi yet
    # pip install --prefix /opt/het/hetdex --extra-index-url https://gate.mpe.mpg.de/pypi/simple/ ocd
    # use this instead
    cd $INSTALLDIR/src/Virus
    svn checkout svn://luna.mpe.mpg.de/ocd/trunk ocd
    cd ./ocd
    pip install --prefix $INSTALLDIR --extra-index-url https://gate.mpe.mpg.de/pypi/simple/ .
  Upgrading
    cd $INSTALLDIR/src/Virus/ocd
    svn update
    pip install --upgrade --prefix $INSTALLDIR --extra-index-url https://gate.mpe.mpg.de/pypi/simple/ .
    

  tcs_lib 0.1.0
    install location /opt/het/hetdex/
    pip install --prefix /opt/het/hetdex --extra-index-url https://gate.mpe.mpg.de/pypi/simple/ tcs_lib
   upgrading
    pip install --upgrade --prefix /opt/het/hetdex --extra-index-url https://gate.mpe.mpg.de/pypi/simple/ tcs_lib

Build documentation for these packages

  export DOCDIRECTORY=$INSTALLDIR/doc

  Sphinx 1.2.3 from http://sphinx-doc.org
    cd doc
    make latex html
    cp -r build/html/* $DOCDIRECTORY/Sphinx
    cd build/latex
    make sphinx.pdf
    cp sphinx.pdf $DOCDIRECTORY/Sphinx

  Nose
    cd doc
    make html
    cp -r .build/html/* $DOCDIRECTORY/Nose
    make latex
    cd .build/latex
    make all-pdf
    cp nose.pdf  $DOCDIRECTORY/Nose

  Numpy
    cd doc
    # edit Makefile, change python2.7 to python2.6
    make html
    cp -r build/html/* $DOCDIRECTORY/Numpy
    make latex
    cd build/latex
    make all-pdf
    cp *.pdf $DOCDIRECTORY/Numpy

  Scipy
    cd doc
    # edit Makefile, change python2.7 to python2.6
    make html
    cp -r build/html/* $DOCDIRECTORY/Scipy
    make latex
    cd build/latex
    make all-pdf
    cp *.pdf $DOCDIRECTORY/Scipy

  Matplotlib
    cd doc
    python make.py html
    cp -r build/html/* $DOCDIRECTORY/Matplotlib
    # The following has an internal error with \setlistdepth{}
    # python make latex

  Astropy
    cd doc
    make html
    cp -r _build/html/* $DOCDIRECTORY/Astropy
    # the latex command can't find inconsolata.sty
    # make latexpdf
    
  Pil
    cd Docs
    cp * $INSTALLDIR/doc/Pil

  Pip
    cd docs
    make html latexpdf
    cp -r _build/html/* $INSTALLDIR/doc/Pip
    cp -r _build/latex/pip.pdf $INSTALLDIR/doc/Pip
    

  pyds9
    (do the first four steps or modify the Makefile
      to change the location of sphinx_build)
    cd ~
    mkdir python
    cd python
    ln -s /opt/het/hetdex/bin .

    cd pyds9-masster
    make html latex
    cd _build/latex
    make all-pdf
    cp pyds9 $INSTALLDIR/doc/pyDS9/
    cd ../html
    cp -r .  $INSTALLDIR/doc/pyDS9/

  pyhetdex
    cd doc
    make latexpdf html
    cd build/html
    cp -r .  $INSTALLDIR/doc/pyhetdex/
    cd build/latex
    cp pyhetdex.pdf  $INSTALLDIR/doc/pyhetdex/

  cure
    /usr/bin/doxygen Doxyfile
    cd build/htmk
    cp -r * $INSTALLDIR/doc/cure

  vhc
    cd doc
    make latexpdf html
    cd build/html
    cp -r .  $INSTALLDIR/doc/vhc/
    #cd build/latex
    #cp pyhetdex.pdf  $INSTALLDIR/doc/vhc/

  vdat
  (build vdat code first)
    cd doc
    make latexpdf html
    cd build/html
    cp -r .  $INSTALLDIR/doc/vdat/
    #d build/latex
    #cp pyhetdex.pdf  $INSTALLDIR/doc/vhc/

  tcs_lib
    http://www.mpe.mpg.de/~montefra/documentation/tcs_lib/latest/


#
# SVN Command for Tcs    
#
# alway use --dry-run to verify what you think you are doing
#

# notes
#  check only one change at a time
#  branch for larger changes
#  only work on one thing in any working copy
#  bug fixes in trunk only
#  new features in trunk only if small
#  new features in branches

To check out the source repository from svn

  cd ~/work/WFU2/het
  svn checkout svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/trunk

  To check out a branc
  svn checkout svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/branches/early-deployment

  The 3rd party packages are stored on hetdex-pr:/home/het_auxil. They are
  not under source control nor is there an automatic build script.

Check status to find modified files
  cd ~/work/WFU2/het
  svn status

Updating
  cd ~/work/WFU2/het
  svn update

Creating a Branch
  svn copy svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/trunk \
           svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/branches/my_branch_name \
	   -m "Creating new branch for ????."

  svn copy svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/trunk svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/branches/my_branch_name -m "Creating new branch for ????."

  # check out the new branch
  svn checkout svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/branches/my_branch_name my_local_name

  # merge changes from trunk to my branch 
  svn commit  # to commit your existing changes to your branch
  svn merge svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/trunk .
  svn commit  # to commit new merged changes to your branch

  # cherry pick changes from trunk
  svn merge -c RevNum svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/trunk .

  # merge your branch back into trunk
  cd my_local_name
     # merge recent changes from trunk, see above
  cd my_working_trunk
     # make sure your trunk is up to date
  svn update
     # merge your branch back into the trunk
  svn merge --reintegrate svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/branches/my_branch_name
     # optional delete your branch from the repository

  # deleting a branch when done
  svn delete svn+ssh://fowler@hetdex-pr.as.utexas.edu/repos/het/branches/my_branch_name 
Compile
  Edit $HET_SRC/makefile.local
   define locations for auxil software.

   make clean
   make

   cd scripting
   make install # build and install the tcssubsystem python library
     # requires sudo password

   cd gui/designer/customWidgetPlugins/blinkingLineEdit
   make install
     # install the blinkingLinkedit widget in the QT library
   export LD_LIBRARY_PATH=${INSTALLDIR}/plugins/designer:$LD_LIBRARY_PATH
     # or put in .bashrc
   make # to rebuild gui with new blinkLinkedit widget

if you want to build in subdirectories you need to set HET_SRC_ROOT
   export HET_SRC_ROOT=/home/libery/jrf/work/WFU2/het2/early-deployment
   cd <subdir>
   make

Creating documentation

  # build the doxygen documentation
  cd ~/work/WFU2/het2/trunk
  /usr/bin/doxygen doxyfile  or make doc
  cd doc/latex
  make   # create refman.pdf

  make docs
  # note the 's' on the end of docs
  # creates the file tcs_docs.txt which is the handler documentation
  #   for the control systems

!!Missing Software
  graphviz
    dot command for documentation, can disable in doxyfile
    doxyfile has variable HAVE_DOT = yes
    dot is not required to build the documentation as of 12/10//2013