LRS2 Operating Procedures

The LRS2 instrument is controlled from mcs which issues commands to lrs2.

Beginning of night or upon LRS2 services restart

Starting the Servers

The camra_server and data transfer (called pivot) should be running at all times on lrs2, thus no action from the RA should be required.

If you want to check on these servers try:

<user>@vdas ~]$ chksys
hetdex   24118     1  0 Sep06 ?        00:00:49 /opt/het/hetdex/bin/lrs2_server --named-route tcp://192.168.66.99:30000 -c /opt/het/hetdex/etc/conf/lrs2_server.conf
hetdex   20051     1  0 Sep06 ?        00:00:04 /opt/het/hetdex/bin/tcs_proxy --proxy-route tcp://127.0.0.1:39999 --proxy-command /opt/het/hetdex/bin/proxy-pivot-lrs2-data

If you do need to restart the lrs data acquisition server and the temperature monitor please follow the LRS2 Start Camra Starting LRS2/VDAS software and monitor procedure.

Setting up monitoring

It is advisable to watch the lrs2_server.log located on the lrs2 machine in /var/log/tcs_logs/lrs2/lrs2_server.log. This can be done with

tail -f /var/log/tcs_logs/lrs2/lrs2_server.log

On another terminal, in a different directory, preferably /home/mcs/astronomer (to avoid mixed calls to the db), start a monitor that shows tcs and pfip activity:

monitor --verbose -T -P --key-filter 'log_[^d].*' --log-print

(Note the different filter because of excluding debug messages.)

Ion Pump and Vacuum Gauge

The easiest way to start and stop the IP and VG is with ipvg on or ipvg off

To see the lower level level commands:

From the RA launcher under Engineering bring up the APC gui.

At the start of the night turn off the ION Pump and Vacuum Gauge for both LRS2-B and LRS2-R. To turn off the IONPump just hit the APC green light for those two IONPumps to turn them red. To disable the two VG you must disable them from the command line with

syscmd -l 'disable_pressure_gauges()'

you will not see any feed back in the lrs2logs and a normal response to the command line looks like:

[astronomer@mcs ~]$ syscmd -l 'disable_pressure_gauges()'
{u'__ack': u'false',
 u'__async': u'false',
 u'__data': u'false',
 u'__data_time': u'1484447502.964383517',
 u'__done': u'true',
 u'__effective_id': 338,
 u'__error': u'false',
 u'__handler': u'disable_pressure_gauges',
 u'__origin': u'tcsutils.pyc_lrs215227.mcs.15227.0327b23c6->tcp://192.168.66.15:31300',
 u'__original_id': 2,
 u'__pid': 2835,
 u'__wire_time': u'1484447502.964384730'}

A 60 sec dark frame is really the only way to check this. See the troubleshooting section to restart VG at the end of this section.

If you fail to turn off these you will have extra light evident in the dark frames and science frames. The image below shows the state of the APC LRS2 column when done.

Calibrations

Calibration Script

A prototype calibration script is available that can calibrate VIRUS and LRS2. This script attempts to expose all known supported FCU configurations as well as darks and biases.

Almost all of the time the command below is what we should do for the standard cals.

cal lrs2 -l hg cd-a fear -c qth ldls_long -b 11 - take about 37 minutes (not verified recently)

To get LRS2 and VIRUS biases in parallel use

cal lrs2 -pb -b 20 - about 14 minutes

Here is what I do for the 1st stack of the night:

cal lrs2 -l hg cd-a fear -c qth ldls_long -B - takes 31 minutes (as of 12 Nov 2018)

and if there is time or at some point of the night do:

cal lrs2 -pb -b 20 - takes about 14 minutes

NB

• in case of any problem try to get hardware status of instruments. if got problem try to kill and start camra_server on lrs2
• adjust number of line and continuum lamps (L and C) and obs number as required

The above will expose the lamp1a lamp1b lamp5b ldls (Hg Cd Qth ldls) as observation 1 (and subsequent numbers) and take 20 biases but 0 darks at the end. This is the standard calibration sequence for LRS2.

Calling the script with --help produces more information:

   cal lrs2 --help

The list of lamp combinations is spelled out in a table at the beginning of the script. That table also sets the exposure times and warmup times for the lamps.

Use the --dry-run option to see what the script would do given some options without actually executing any commands on the hardware.

For a standard (default) calibration set, the directory structure for your images will be something like:

lrs20000002    Hg     red,blue x3 each
lrs20000003    Cd     blue x3
lrs20000004    FeAr   red x3
lrs20000005    Qt     red x5
lrs20000006    ldls   blue x3
lrs20000007    bias   exp00-20

You should use lrs2view to inspect at least one image per set.

enable_shutter

Stopping Calibrations

If you wish to stop the calibrations elegantly you should use the command sc There is a help for sc with the command sc -h. If you want to stop cal.py in an exposure immediately use the command sc -i.

Twilight

Twilights should be taken with type "twi". If you use GC2 with the B filter and a 0.1 second exposure when you get a count of 30,000 ADU then you are ready to take a 15 second twilight:

vlexp -i lrs2 -pobj twilight -texp 15 -t twi

Get help on ldas command

• The syscmd mode can be used to display help information on ldas commands. Here are some examples:

  syscmd -l 'help("abort_exposure")'
  syscmd -l 'help("wait_for_readout")'
  syscmd -l 'help("wait_for_write")'
  syscmd -l 'help("wait_for_shutter_close")'
  

• Suppose you only remember part of the command name:

  syscmd -l 'help()' | grep wait
  

• Suppose you want to be notified when the shutter closes:

  syscmd -l 'wait_for_shutter_close(observation=8001, exposure=1  )'
  beep
  beep
  beep
  

The command will not return until the shutter closes. at which point in the example above you'll hear three beeps.

Setup LRS2 Observations

The majority of our LRS2 targets are qrouped targets requiring spectra in both the LRS2-B and LRS2-R IFUs. Hence, we desire probe stars that will be valid for both of these sky positions. Furthermore, we desire at least one GC star and one WF star. The lrs2stars routine allows us to locate stars for the GC and WF probes that will allow offsets between the LRS2-R abd LRS2-B positions on the sky.

• For a Main Queue target (e.g. 18-1) you must know the index number for the LRS2-R target AND the index for the LRS2-B target.

• For fixed catalogs (e.g. specphot, telluric, etc...) you must know only the one index number assigned to each target.

• In the example below I set up probe stars for index=273 in the main catalog for trimester 1 of 2018. The east (E) track is setup in this example. Note that the index=273 entry is the LRS2-R setup (ifu=066) and we will need to know the corresponding index of the LRS2-B target to be jumped to (index=272 for ifu=056).

lrs2stars 18-1 273 E RB

• To get more detailed descriptions for using lrs2stars:

lrs2stars --help

• You can also read ​file://home/mcs/sco/scohtm/scocodes/lrs2stars/ if running on mcs.

• Note that probes can only get within ~15deg of each other. choose stars to permit that.

• Use the 3 lines (cp, load_traj, mgp) from its output, plus the metadata, WF exptime, and 2 ACQ lines.
  • after iexp of the first, apply: mgp -toff=BR then use target_setup commands for metadata ONLY for the R (if any star was unavailable for any probe, use mgp -u gc2 to un-set it, e.g., GC2). After mgp offset, ACAM will verify position. may need to re-center WFS.

• Take the lrs2 observations using "iexp" (see next section).

• View images with lrs2view (must be done on lrs2):

  cd /hetdata/data/NWD/lrs2/lrs20001001/exp01/lrs2     ! to get to data directory
  lrs2view 56       ! to view LRS2-B images
  lrs2view 66       ! to view LRS2-R images
  

• To move from LRS2-B to LRS2-R, we use offset_trajectory:

B2R:  mgp -toff=BR 
R2B:  mgp -toff=RB

• Finally, if the data transfer for lrs2 does not complete, you should log in as hetdex and do:

data_check lrs2

Taking Science Exposures

A target must be placed on either the lrs2b or lrs2r markers for the lrs2 to see them. Once the object is on the correct marker and guiding has been achieved on the guide probe be sure to have the TO retract the acquisition camera. It is critical that you log which unit of lrs2 the target is on, since asking for an lrs2 exposure will read out both the lrs2b and lrs2r. It is also important to note the observation number that is currently being used. Do not over-write the beginning of the night calibrations. Perhaps starting the science numbering with 1000 and the start of the night calibration numbers with 100 will avoid this problem?

::::::::::::::::::::: old approach :::::::::::::::::::::::

Be sure to have the TO remove the retract the ACQ

It is important to tell lrs2 that it is going to have control of the PFIP shutter. This is done by executing the following on lrs2 (logged in as yourself):

syscmd -l 'enable_shutter()'

There are many ways to take an exposure; the simplest uses the vlexp command:

vlexp -i lrs2 -pobj mrk110 -texp 200 -B

vlexp will take parallel exposures buy default. Exceptions are exposure time less then 300s or if indicate argument "-np"

or using lower level commands:

# LRS2:
syscmd -l 'expose( seconds=20, x_binning=1, y_binning=1, type="sci", object="mrk110", observation=1234, exposure=1 )'

The exposure command will return a prompt once the readout has started. The output FITS files will be available 40-50 seconds after that, depending on binning and disk I/O. The return string of the expose command provides the output directory for that exposure.

The exposure time is in the "seconds" parameter (minimum of 15).

::::::::::::::::::::: new approach :::::::::::::::::::::::

Use command iexp. Here are the logic:

• RA sends new target and run iexp as soon as TO press go next
• RA check command and press Enter and iexp sit wating for "Setup complete" to happen
• TO setup on target and having ACQ images saving all the time during setup
• As soon as it's ready TO press "Setup complete" and iexp starts immediately
• iexp
  • retract ACQ
  • start saving images on probes and wfs's
  • start exposure
• at the end of exposure iexp
  • stop saving probes images
  • set exp time for wfs's to 5s
  • cansel setup
  • insert ACQ

At this point system is ready for new iexp command

Observation and exposure determine the directories under which the data are grouped. An example output directory structure might be

/hetdata/data/NWD/lrs2/lrs20000023/exp01/
                             /exp02/

So here two exposures are grouped under the observation number 23 taken on 20160309.

The combination of observation/exposure must be unique for the UTC date. The agreed upon types are "sci", "cmp", "zro", "flt", "drk", "tst".

To stop and readout an exposure use the command syscmd -l 'abort_exposure()'

Doing Parallel Science with LRS as primary

To get help on this command try vlexp --help for an example with the LRS2 as primary

vlexp -i lrs2 -t sci -texp 600 -pobj Myobject2 -B

NOTE: There will be no dithering for this type of exposure.

To stop and readout both the LRS2 and VIRUS do syscmd -l 'abort_exposure()' and syscmd -V 'abort_exposure()' quickly back to back.

End of Night Procedures

Turn on the LRS2 VG and LRS2 IP and look at the LRS2 log to make sure that the VG connected. Wait 10 minutes and then take a reading and record that into the RA logs. The APC gui should look like the following:

Another way to turn the VG,IP on and record pressures:

 ipvg on stat

or

 ipvg on
 wait 10 minuts and print cryo pressures:
 syscmd -l 'get_cryo_pressure( update_state="true" )'

Record the final LRS2 temperatures into the RA log.

At the present time all data transfers are automated and no interaction from the RA is required.

Troubleshooting

Checking Hardware Status

syscmd -l 'get_hardware_status( update_state="true" )'

The status may be queried mid-exposure by omitting the update_state parameter.

Typical problems

• For a constant value in an amp you only need to restart the service.
• For a readout that looks like hash or huge noise you need to restart the service and the controller
• For an error in the /var/log/tcs_logs/lrs2/lrs2_server.log that includes an error with "MUX"; you need to restart the service, controller and mux.

Restarting the Controllers and Muxes

When CAMRA crashes, it is likely due to a controller going offline. That will not be seen in the log messages displayed from the monitor and, in the case described here, if it happens in the middle of an exposure it can leave a connected client script hanging waiting for the response to a command (i.e. expose, get_ccd_temp, etc). The *only* place you can see what is going on in this case is by tailing the /var/log/tcs_logs/lrs2/lrs2_server.log file on lrs2. You will see that the log ends with a string that includes "CArc" and "exception" in the case of a hardware failure.

When there is a problem with the hardware, there should be no attempt to bring lrs2 back up without power cycling the controllers. I don't think this has to include the muxes but here is a prescription for how to restart lrs2 from scratch:

1. From the RA launcher under Operations and Servers and LDAS Camra select stop. This is equivalent to the old way "From your account on lrs2: If running, stop ldas sudo service ldas stop "
   
2. On the APC GUI Power off hardware LRS2 controller
   
3. Turn off the LRS2MUX on the APC GUI
   
4. On the APC GUI Power on LRS2 controller
   

5. Wait 30 seconds then on the APC GUI Power on LRS2MUX
   

6. From the RA launcher under Operations and Servers and LDAS Camra select start. This is equivalent to the old way "From your account on lrs2: If running, stop ldas sudo service ldas start "
   
7. Watch the /var/log/tcs_logs/lrs2/lrs2_server.log for messages about Accepting commands tail -f /var/log/tcs_logs/lrs2/lrs2_server.log
   
8. then take a test exposure cal lrs2 -b 1 -d 0 -o 5555
9. Check for 0 bias frames with a test lrs2 exposure

Pivot script problem

If pivot is not running and ldas_pivot.log have message like:

Failed to bind to tcp://127.0.0.1:39999: Address already in use

use following recipe (from RAlauncher)

• stop lrs2Server
• wait 1 minute
• start lrs2Pivot
• wait until message 'Accepting commands ...' will appear in pivot log
• start lrs2Server

• check that lrs2DataTransfer service are running (through RA launcher)
  It should report something like : lrs2DataTransfer (pid 36304) is running...
• if not restart it through RA launcher

If the ramdisk fills up (probably because pivot is not running) then try the following:

• stop lrs2Server
• wait 1 minute
• start lrs2Pivot
• check the lrs2_pivot log to make sure it is running properly (use RAlauncher)
• start lrs2Server
• as hetdex on lrs2 execute dt_fix lrs2 this will start moving data off the ramdisk but might take 10 minutes.

Restarting the Vacuum Gauge

If you start the VG and get a very slow response back and perhaps the error:

[astronomer@mcs ~]$ syscmd -l 'enable_pressure_gauges()'
Traceback (most recent call last):
  File "/opt/het/hetdex/bin/syscmd", line 163, in <module>
    result = eval( 'cli.' + command )
  File "<string>", line 1, in <module>
  File "<string>", line 3, in enable_pressure_gauges_method
pytcs.error: remote exception: Failed to connect to the pressure gauge '192.168.66.151:8000'

and in the lrs2 logs:

Failed to connect to the pressure gauge '192.168.66.151:8000'

Jason suggests trying the following:

apcCmd off LRS2R-VacGaugeCntrlr
apcCmd off LRS2B-VacGaugeCntrlr
sleep 30
apcCmd on LRS2R-VacGaugeCntrlr
apcCmd on LRS2B-VacGaugeCntrlr
sleep 120
syscmd -l 'get_cryo_pressure( cntl=0, update_state="true" )'

A couple things to note:

• You may get an error on this last command in the LRS2 log that says ERROR [camra_hardware.cpp :checkReply :1460] - Pressure reading failed with error return code '4'. This error indicates that the pump is still warming up.
• I never called enable_pressure_gauges(). Since camra thinks they are already enabled it will attempt to reconnect to the gauge on the 5 minute polling interval, or when calling one of the status handlers with update_state="true". Calling enable_pressure_gauges() isn't a problem, but timing might be.
• I let the polling cycle complete one iteration prior to calling get_cryo_pressure(). The logs show the reconnect and the first sample with the invalid status of 4 (the gauge is still warming up), see below.
• I think the timing between power cycling things through the apc and attempting to reconnect to them may be the key problem you are having... but I don't have any concrete rules to follow.

Here are the older notes:

Then try the disabling the VG again, cycling power to the VG through the APC gui and re-enabling the VG again:

syscmd -l 'disable_pressure_gauges()'
apcCmd off LRS2B-VacGaugeCntrlr          #  or just click on the APCgui but always check that it turned off in the APCgui
apcCmd off LRS2R-VacGaugeCntrlr          #  or just click on the APCgui but always check that it turned off in the APCgui
Current lore: restart lrs2 server    SCO, Sep12,2018
Second piece of lore:   do not do the next 3 steps --- JR,RB Sep12,2018 
apcCmd on LRS2B-VacGaugeCntrlr          #  or just click on the APCgui but always check that it turned off in the APCgui
apcCmd on LRS2R-VacGaugeCntrlr          #  or just click on the APCgui but always check that it turned off in the APCgui
syscmd -l 'enable_pressure_gauges()'

in the LRS2 log you should see

lrs2-b-vg-cntrl.as.utexas.edu:192.168.66.151:8000 connected
lrs2-r-vg-cntrl.as.utexas.edu:192.168.66.152:8000 connected

remember to wait 10 minutes before getting an accurate VG reading.

Aug9,2018 == We have been having problem communicating with the VG. This is indicated by errors when we run "ipvg off". We see complaints about being unable to reach an IP address. The fix seems to be to manual cycle VG power in the tcs gui. I.E. we turn the VG lights red then green. After that I was doing a "syscmd -l 'enable_pressure_gauges()' after the VG light was green. This seemed to put the system back into nirmal state. (SCO)

Hardware Changes

If one of the electronics day staff changes a controller a change must be made to the configuration files. For both LRS2 and VIRUS this must be done by either Phillip, Jason or Jim so call them at home at any hour needed to get the instrument running.

Setting up stars to avoid a bad or dead amp

If one of the LRS2 amps is dead then you can try to place a point source on the other half of the IFU as to avoid that amp. For example if you want to avoid the LU or RL amps then you will need to offset the star by +1.5 arcseconds in tracker Y from the ideal setup position or setup +5.5 pixels to larger X values on the ACAM. Conversly if you want to avoid the RU or LL amps then you will need to offset the star by -1.5 arcseconds in tracker Y or setup -5.5 pixels to smaller X values on the ACAM.

Additional Help

Help for the complete set of CAMRA commands can be obtained by issuing a 'help()' command to the CAMRA subsystem.

syscmd -l 'help()'