= Creating a system monitor =

A monitor can be run from inside the TCS gui or from the command line in htcs, but to avoid
cross purposes with TO, run it preferably from /home/mcs/astronomer.

In the following example on is run for the TCS (note the -T) but it excludes the debug commands (note the ^d):

{{{ monitor --verbose -T --log-print --key-filter='log_[^d].*' }}} 

In the following example you can get the WFS seeing estimate from pas:

{{{monitor --verbose -p --key-filter='metrology_data' --source='WFS*' | grep "time\|seeing\|\.metrology" }}}

== Log files ==

Perhaps more useful than the db are the log files.   These can be accessed during the night and include the current night a usually about a month of order logs.   There are a number of different logs on different machines:
  * TCS logs are located on htcs in /var/log/tcs_logs/subsystem_name/
  * TCS_GUI logs are located on jove or mcs in /var/tcs_logs/
  * VIRUS and LRS logs are located on vdas and lrs2 in /var/log/tcs_logs/instrument_name/
  * Htopx logs are located in /data1/nossy/www/html/noss/htopx3/log/htopx.log


The TCS logs are split into the different sub-systems: legacy, pfip, pas, tcs, and tracker.   There are several ways one might use these.  Note that to run this command you must be logged into htcs (e.g. ssh -X sco@htcs).

   * For example to look at the pfip probe motions to check for the "move complete" before making any further moves.   This would be done as {{{tail -F tcs_server.log | grep pfip}}}
   * One can also look for a "go_next" or "load_trajectory" inside of the tcs_server logs.
   * Be sure to be logged into htcs.

One could look at the TCS GUI logs to see what buttons were pushed to get an idea if a problem was a human error or a software problem.   The htopx_logs are useful for looking to see when/if targets were incremented.

== Nightly Databases ==

Nightly databases can be found in /opt/het/hetdex/logs/Mon/YYYY/MM/ where YYYY and MM are the year and month.  Data files are written at around 18 UT and are off by one UT date.   For example /opt/het/hetdex/logs/Mon/2017/05/20170513T180005.db is the database for the night of 20170514.

You can extract a text version of the db with the python code:  db2log.py


== Plotter ==

Here is an example of plotting guider1 metrology x vs. y: {{{tcs_plotter --named-route "tcp://192.168.66.99:30000" --plotter-route "" --endpoint-system tcs --system tcs --source Guider1 --key measurement -x 'focal_plane.dx_asec' -y 'focal_plane.dy_asec'}}}

Here is another example {{{ tcs_plotter --named-route "tcp://192.168.66.99:30000" --plotter-route "" --endpoint-system tcs --system tcs --source root --key ra_dec -x 'ra' -y 'dec'}}}

== Other useful monitors ==

Looking at the DMI and WFS data:

{{{monitor --verbose -T --source-filter 'WFS*|DMI' --key-filter measurement }}}

== working with the htopx and night report databases ==

There are many databases here.    some of the more useful ones are:
  * plandb_latest_cur (this holds the most recent update for any record in the current queue)
  * plandb (holds all changes and version so allows you to look at history)
  * plandb_latest_new (this is for the new trimester)
  * nightreport_ra_targets
  * tacs
  * 


It is useful to know what fields can be queried with a mysql query in these databases.   To get a list try:  {{{ esql htopx -e "describe fullqueue"  }}} or {{{ mysql -uhtopx -p... htopx -e "describe fullqueue" }}}