Creating a system monitor

A monitor can be run from inside the TCS gui or from the command line in htcs, but to avoid cross purposes with TO, run it preferably from /home/mcs/astronomer.

In the following example on is run for the TCS (note the -T) but it excludes the debug commands (note the ^d):

monitor --verbose -T --log-print --key-filter='log_[^d].*'

In the following example you can get the WFS seeing estimate from pas:

monitor --verbose -p --key-filter='metrology_data' --source='WFS*' | grep "time\|seeing\|\.metrology"

Log files

Perhaps more useful than the db are the log files. These can be accessed during the night and include the current night a usually about a month of order logs. There are a number of different logs on different machines:

• TCS logs are located on htcs in /var/log/tcs_logs/subsystem_name/
• TCS_GUI logs are located on jove or mcs in /var/tcs_logs/
• VIRUS and LRS logs are located on vdas and lrs2 in /var/log/tcs_logs/instrument_name/
• Htopx logs are located in /data1/nossy/www/html/noss/htopx3/log/htopx.log

The TCS logs are split into the different sub-systems: legacy, pfip, pas, tcs, and tracker. There are several ways one might use these. Note that to run this command you must be logged into htcs (e.g. ssh -X sco@htcs).

• For example to look at the pfip probe motions to check for the "move complete" before making any further moves. This would be done as tail -F tcs_server.log | grep pfip
• One can also look for a "go_next" or "load_trajectory" inside of the tcs_server logs.
• Be sure to be logged into htcs.

One could look at the TCS GUI logs to see what buttons were pushed to get an idea if a problem was a human error or a software problem. The htopx_logs are useful for looking to see when/if targets were incremented.

Nightly Databases

Nightly databases can be found in /opt/het/hetdex/logs/Mon/YYYY/MM/ where YYYY and MM are the year and month. Data files are written at around 18 UT and are off by one UT date. For example /opt/het/hetdex/logs/Mon/2017/05/20170513T180005.db is the database for the night of 20170514.

You can extract a text version of the db with the python code: db2log.py

Plotter

Here is an example of plotting guider1 metrology x vs. y: tcs_plotter --named-route "tcp://192.168.66.99:30000" --plotter-route "" --endpoint-system tcs --system tcs --source Guider1 --key measurement -x 'focal_plane.dx_asec' -y 'focal_plane.dy_asec'

Here is another example tcs_plotter --named-route "tcp://192.168.66.99:30000" --plotter-route "" --endpoint-system tcs --system tcs --source root --key ra_dec -x 'ra' -y 'dec'

Other useful monitors

Looking at the DMI and WFS data:

monitor --verbose -T --source-filter 'WFS*|DMI' --key-filter measurement

working with the htopx and night report databases

There are many databases here. some of the more useful ones are:

• plandb_latest_cur (this holds the most recent update for any record in the current queue)
• plandb (holds all changes and version so allows you to look at history)
• plandb_latest_new (this is for the new trimester)
• nightreport_ra_targets
• tacs
• ephemerides
• nightreport_ra_programs

It is useful to know what fields can be queried with a mysql query in these databases. To get a list try: esql htopx -e "describe fullqueue" or mysql -uhtopx -p... htopx -e "describe fullqueue"