= LRS2 Operating Procedures =

The LRS2 instrument is controlled from mcs which issues commands to lrs2.

== Beginning of night or upon LRS2 services restart == #Startup

=== Starting the Servers ===

The camra_server and data transfer (called pivot) should be running at all times on lrs2, thus no action from the RA should be required.  

If you want to check on these servers try:

{{{ 
<user>@vdas ~]$ chksys
hetdex   24118     1  0 Sep06 ?        00:00:49 /opt/het/hetdex/bin/lrs2_server --named-route tcp://192.168.66.99:30000 -c /opt/het/hetdex/etc/conf/lrs2_server.conf
hetdex   20051     1  0 Sep06 ?        00:00:04 /opt/het/hetdex/bin/tcs_proxy --proxy-route tcp://127.0.0.1:39999 --proxy-command /opt/het/hetdex/bin/proxy-pivot-lrs2-data

}}}

If you do need to restart the lrs data acquisition server and the temperature monitor please follow the 
[wiki:StartCamra LRS2 Start Camra] Starting LRS2/VDAS software and monitor procedure.


=== Setting up monitoring ===


It is advisable to watch the lrs2_server.log located on the lrs2 machine in /var/log/tcs_logs/lrs2/lrs2_server.log.   This can be done with 
{{{
tail -f /var/log/tcs_logs/lrs2/lrs2_server.log
}}}


On another terminal, in a different directory, preferably /home/mcs/astronomer (to avoid mixed calls to the db), start a monitor that shows tcs and pfip activity: 

{{{
#!div style="font-size: 80%"
  {{{#!bash
  monitor --verbose -T -P --key-filter 'log_[^d].*' --log-print
  }}}
}}}

(Note the different filter because of excluding debug messages.)

=== Ion Pump and Vacuum Gauge ===


The easiest way to start and stop the IP and VG is with {{{ ipvg on }}} or {{{ ipvg off }}}

To see the lower level level commands:

From the RA launcher under Engineering bring up the APC gui.

At the start of the night turn off the ION Pump and Vacuum Gauge for both LRS2-B and LRS2-R.  To turn off the IONPump just hit the APC green
light for those two IONPumps to turn them red.   To disable the two VG you must disable them from the command line with 

{{{ syscmd -l 'disable_pressure_gauges()' }}}  

you will not see any feed back in the lrs2logs and a normal response to the command line looks like:

{{{
[astronomer@mcs ~]$ syscmd -l 'disable_pressure_gauges()'
{u'__ack': u'false',
 u'__async': u'false',
 u'__data': u'false',
 u'__data_time': u'1484447502.964383517',
 u'__done': u'true',
 u'__effective_id': 338,
 u'__error': u'false',
 u'__handler': u'disable_pressure_gauges',
 u'__origin': u'tcsutils.pyc_lrs215227.mcs.15227.0327b23c6->tcp://192.168.66.15:31300',
 u'__original_id': 2,
 u'__pid': 2835,
 u'__wire_time': u'1484447502.964384730'}
}}}

A 60 sec dark frame is really the only way to check this.  See the [#PRVG troubleshooting] section to restart VG at the end of this section.  

If you fail to turn off these you will have extra light evident in the dark frames and science frames. The image below shows the state of the APC LRS2 column when done. 

[[Image(APCLRS2startofnight.png)]]




== Calibrations == #cals

=== Calibration Script

A prototype calibration script is available that can calibrate VIRUS and LRS2.  This script attempts to expose all known supported FCU configurations as well as darks and biases.  

Almost all of the time the command below is what we should do for the standard cals. 

{{{cal lrs2  -l hg cd-a fear -c qth ldls_long -b 11}}}  - take about 37 minutes (not verified recently)

To get LRS2 and VIRUS biases in parallel use

{{{cal lrs2 -pb -b 20 }}} - about 14 minutes

Here is what I do for the 1st stack of the night:

{{{ cal lrs2  -l hg cd-a fear -c qth ldls_long -B  }}} - takes 31 minutes (as of 12 Nov 2018)

and if there is time or at some point of the night do:

{{{cal lrs2 -pb -b 20 }}} - takes about 14 minutes


NB
- in case of any problem try to get hardware status of instruments.
  if got problem try to kill and start camra_server on lrs2
- adjust number of line and continuum lamps (L and C) and obs number as required

The above will expose the lamp1a lamp1b lamp5b ldls (Hg Cd Qth ldls) as observation 1 (and subsequent numbers) and take 20 biases but 0 darks at the end. This is the standard calibration sequence for LRS2.

Calling the script with --help produces more information:

{{{
#!div style="font-size: 80%"
  {{{#!bash
     cal lrs2 --help
  }}}
}}}

The list of lamp combinations is spelled out in a table at the beginning of the script. That table also
sets the exposure times and warmup times for the lamps.

Use the --dry-run option to see what the script would do given some options without actually executing any commands on the hardware.

For a standard (default) calibration set, the directory structure for 
your images will be something like:
{{{
lrs20000002    Hg     red,blue x3 each
lrs20000003    Cd     blue x3
lrs20000004    FeAr   red x3
lrs20000005    Qt     red x5
lrs20000006    ldls   blue x3
lrs20000007    bias   exp00-20
}}}
You should use lrs2view to inspect at least one image per set. 

enable_shutter

==== Stopping Calibrations ====

If you wish to stop the calibrations elegantly you should use the command {{{sc}}}   There is a help for sc with the command {{{sc -h}}}.   If you want to stop cal.py in an exposure immediately use the command {{{sc -i}}}.

=== Twilight ===

Twilights should be taken with type "twi".   If you use GC2 with the B filter and a 0.1 second exposure when you get a count of 30,000 ADU then you are ready to take a 15 second twilight:

{{{ vlexp -i lrs2 -pobj twilight -texp 15 -t twi }}}

== Get help on ldas command ==
* The syscmd mode can be used to display help information on ldas commands. Here are some examples:
{{{
syscmd -l 'help("abort_exposure")'
syscmd -l 'help("wait_for_readout")'
syscmd -l 'help("wait_for_write")'
syscmd -l 'help("wait_for_shutter_close")'
}}}

* Suppose you only remember part of the command name:
{{{
syscmd -l 'help()' | grep wait
}}}

* Suppose you want to be notified when the shutter closes:
{{{
syscmd -l 'wait_for_shutter_close(observation=8001, exposure=1  )'
beep
beep
beep
}}}
The command will not return until the shutter closes. at which point in the example above you'll hear three beeps. 

== Setup LRS2 Observations == #setup
The majority of our LRS2 targets are qrouped targets requiring spectra in both the LRS2-B and LRS2-R IFUs. Hence, we desire probe stars that will be valid for both of these sky positions. Furthermore, we desire at least one GC star and one WF star. The lrs2stars routine allows us to locate stars for the GC and WF probes that will allow offsets between the LRS2-R abd LRS2-B positions on the sky. 

  * For a Main Queue target (e.g. 18-1) you must know the index number for the LRS2-R target AND the index for the LRS2-B target.

  * For fixed catalogs (e.g. specphot, telluric, etc...) you must know only the one index number assigned to each target.

  * In the example below I set up probe stars for index=273 in the main catalog for trimester 1 of 2018. The east (E) track is setup in this example. Note that the index=273 entry is the LRS2-R setup (ifu=066) and we will need to know the corresponding index of the LRS2-B target to be jumped to (index=272 for ifu=056). 

{{{lrs2stars 18-1 273 E RB}}}

 * To get more detailed descriptions for using lrs2stars:

{{{lrs2stars --help}}}

 * You can also read [file://home/mcs/sco/scohtm/scocodes/lrs2stars/] if running on mcs. 
 
 * Note that probes can only get within ~15deg of each other. choose stars to permit that.

 * Use the 3 lines (cp, load_traj, mgp) from its output, plus the metadata, WF exptime, and 2 ACQ lines.
      * after iexp of the first, apply: {{{mgp -toff=BR}}}    then use target_setup commands for metadata ONLY for the R (if any star was unavailable for any probe, use {{{mgp -u gc2}}} to un-set it, e.g., GC2). After mgp offset, ACAM will verify position. may need to re-center WFS.

 * Take the lrs2 observations using "iexp" (see next section). 

 * View images with lrs2view (must be done on lrs2):
{{{
cd /hetdata/data/NWD/lrs2/lrs20001001/exp01/lrs2     ! to get to data directory
lrs2view 56       ! to view LRS2-B images
lrs2view 66       ! to view LRS2-R images
}}}

 * To move from LRS2-B to LRS2-R, we use offset_trajectory:

{{{ 
B2R:  mgp -toff=BR 
R2B:  mgp -toff=RB
}}}

  * Finally, if the data transfer for lrs2 does not complete, you should log in as hetdex and do:

{{{data_check lrs2}}}


== Taking Science Exposures == #obs

A target must be placed on either the lrs2b or lrs2r markers for the lrs2 to see them.  Once the object is on the correct marker and guiding has been achieved on the guide probe be sure to have the TO retract the acquisition camera.  
It is critical that you log which unit of lrs2 the target is on, since asking for an lrs2 exposure will read out both the lrs2b and lrs2r.   It is also important to note the observation number that is currently being used.    Do not over-write 
the beginning of the night calibrations.   Perhaps starting the science numbering with 1000 and the start of the night calibration numbers with 100 will avoid this problem?

::::::::::::::::::::: old approach :::::::::::::::::::::::

'''Be sure to have the TO remove the retract the ACQ'''

It is important to tell lrs2 that it is going to have control of the PFIP shutter.  This is done by executing the following on lrs2 (logged in as yourself):

{{{ syscmd -l 'enable_shutter()'}}}

There are many ways to take an exposure; the simplest uses the vlexp command:

{{{ vlexp -i lrs2 -pobj mrk110 -texp 200 -B }}}

vlexp will take parallel exposures buy default. Exceptions are exposure time less then 300s or if indicate argument "-np" 

or using lower level commands:

{{{
#!div style="font-size: 80%"
  {{{#!bash
  # LRS2:
  syscmd -l 'expose( seconds=20, x_binning=1, y_binning=1, type="sci", object="mrk110", observation=1234, exposure=1 )'
  }}}
}}}



The exposure command will return a prompt once the readout has started.  The output FITS files will be available 40-50 seconds after that, depending on binning and disk I/O.  The return string of the expose command provides the output directory for that exposure.

The exposure time is in the "seconds" parameter (minimum of 15). 

::::::::::::::::::::: new approach :::::::::::::::::::::::

Use command {{{iexp}}}. Here are the logic:
* RA sends new target and run iexp as soon as TO ```press go next```
* RA check command and press Enter and iexp sit wating for "Setup complete" to happen
* TO setup on target and having ACQ images saving all the time during setup
* As soon as it's ready TO press "Setup complete" and iexp starts immediately
* iexp
  * retract ACQ
  * start saving images on probes and wfs's
  * start exposure
* at the end of exposure iexp
  * stop saving probes images
  * set exp time for wfs's to 5s
  * cansel setup
  * insert ACQ
At this point system is ready for new iexp command


Observation and exposure determine the directories under which the data are grouped. An example output directory structure might be

{{{
/hetdata/data/NWD/lrs2/lrs20000023/exp01/
                             /exp02/
}}}

So here two exposures are grouped under the observation number 23 taken on 20160309.

The combination of observation/exposure must be unique for the UTC date. The agreed upon types are "sci", "cmp", "zro", "flt", "drk", "tst".

To stop and readout an exposure use the command {{{syscmd -l 'abort_exposure()'}}}

== Doing Parallel Science with LRS as primary == #parallel


To get help on this command try {{{ vlexp --help }}} for an example with the LRS2 as primary 

{{{ vlexp -i lrs2 -t sci -texp 600 -pobj Myobject2 -B }}}

NOTE:  There will be no dithering for this type of exposure.

To stop and readout both the LRS2 and VIRUS do {{{ syscmd -l 'abort_exposure()' }}} and {{{  syscmd -V 'abort_exposure()' }}} quickly back to back.


== End of Night Procedures == #park


Turn on the LRS2 VG and LRS2 IP and look at the LRS2 log to make sure that the VG connected.  Wait 10 minutes and then take a reading 
and record that into the RA logs.  The APC gui should look like the following:

[[Image(APCLRS2endofnight.png)]]

Another way to turn the VG,IP on and record pressures:
{{{
 ipvg on stat
}}}
or
{{{
 ipvg on
 wait 10 minuts and print cryo pressures:
 syscmd -l 'get_cryo_pressure( update_state="true" )'
}}}

Record the final LRS2 temperatures into the RA log.


At the present time all data transfers are automated and no interaction from the RA is required.

== Troubleshooting == #PR

=== Checking Hardware Status ===

{{{
#!div style="font-size: 80%"
  {{{#!bash
  syscmd -l 'get_hardware_status( update_state="true" )'
  }}}
}}}


The status may be queried mid-exposure by omitting the update_state parameter.


=== Typical problems ===

*  For a constant value in an amp you only need to restart the service.
*  For a readout that looks like hash or huge noise you need to restart the service and the controller
*  For an error in the /var/log/tcs_logs/lrs2/lrs2_server.log that includes an error with "MUX"; you need to restart the service, controller and mux.

=== Restarting the Controllers and Muxes ===

When CAMRA crashes, it is likely due to a controller going offline.  That will not be seen in the log messages displayed from the monitor and, in the case described here, if it happens in the middle of an exposure it can leave a connected client script hanging waiting for the response to a command (i.e. expose, get_ccd_temp, etc).  The *only* place you can see what is going on in this case is by tailing the /var/log/tcs_logs/lrs2/lrs2_server.log file on lrs2.  You will see that the log ends with a string that includes "CArc" and "exception" in the case of a hardware failure.

When there is a problem with the hardware, there should be no attempt to bring lrs2 back up without power cycling the controllers.  I don't think this has to include the muxes but here is a prescription for how to restart lrs2 from scratch:


1. From the RA launcher under '''Operations''' and '''Servers''' and '''LDAS Camra''' select '''stop'''.  This is equivalent to the old way  "From your account on lrs2:  If running, stop ldas {{{ sudo service ldas stop }}}"[[BR]]
2. On the APC GUI Power off hardware LRS2 controller  [[BR]]
3. Turn off the '''LRS2MUX''' on the APC GUI [[BR]]
4. On the APC GUI Power on LRS2 controller [[BR]]
[[Image(LRS2TDK.png)]]
5. Wait 30 seconds then on the APC GUI Power on LRS2MUX [[BR]]
[[Image(LRS2Mux.png)]]
6. From the RA launcher under '''Operations''' and '''Servers''' and '''LDAS Camra''' select '''start'''.  This is equivalent to the old way  "From your account on lrs2:  If running, stop ldas {{{ sudo service ldas start }}}"[[BR]]
7. Watch the /var/log/tcs_logs/lrs2/lrs2_server.log for messages about Accepting commands {{{ tail -f /var/log/tcs_logs/lrs2/lrs2_server.log }}}  [[BR]]
8. then take a test exposure {{{ cal lrs2  -b 1 -d 0 -o 5555 }}}
9. Check for 0 bias frames with a test lrs2 exposure


=== Pivot script problem ===
If pivot is not running and ldas_pivot.log have message like:

Failed to bind to tcp://127.0.0.1:39999: Address already in use

use following recipe (from RAlauncher)
- stop lrs2Server
- wait 1 minute
- start lrs2Pivot
- wait until message 'Accepting commands ...' will appear in pivot log
- start lrs2Server

- check that lrs2DataTransfer service are running (through RA launcher) [[br]]
  It should report something like : {{{ lrs2DataTransfer (pid  36304) is running... }}}
- if not restart it through RA launcher


If the ramdisk fills up (probably because pivot is not running) then try the following:
- stop lrs2Server
- wait 1 minute
- start lrs2Pivot
- check the lrs2_pivot log to make sure it is running properly (use RAlauncher) 
- start lrs2Server
- as hetdex on lrs2 execute {{{dt_fix lrs2}}}  this will start moving data off the ramdisk but might take 10 minutes.



=== Restarting the Vacuum Gauge === #PRVG

If you start the VG and get a very slow response back and perhaps the error:

{{{
[astronomer@mcs ~]$ syscmd -l 'enable_pressure_gauges()'
Traceback (most recent call last):
  File "/opt/het/hetdex/bin/syscmd", line 163, in <module>
    result = eval( 'cli.' + command )
  File "<string>", line 1, in <module>
  File "<string>", line 3, in enable_pressure_gauges_method
pytcs.error: remote exception: Failed to connect to the pressure gauge '192.168.66.151:8000'
}}}

and in the lrs2 logs:

{{{
Failed to connect to the pressure gauge '192.168.66.151:8000'
}}}

Jason suggests trying the following:

{{{
apcCmd off LRS2R-VacGaugeCntrlr
apcCmd off LRS2B-VacGaugeCntrlr
sleep 30
apcCmd on LRS2R-VacGaugeCntrlr
apcCmd on LRS2B-VacGaugeCntrlr
sleep 120
syscmd -l 'get_cryo_pressure( cntl=0, update_state="true" )'
}}}

A couple things to note:

* You may get an error on this last command in the LRS2 log that says '''ERROR [camra_hardware.cpp  :checkReply          :1460] - Pressure reading failed with error return code '4''''.   This error indicates that the pump is still warming up.
* I never called enable_pressure_gauges().  Since camra thinks they are already enabled it will attempt to reconnect to the gauge on the 5 minute polling interval, or when calling one of the status handlers with update_state="true".  Calling enable_pressure_gauges() isn't a problem, but timing might be.
* I let the polling cycle complete one iteration prior to calling get_cryo_pressure().  The logs show the reconnect and the first sample with the invalid status of 4 (the gauge is still warming up), see below.
* I think the timing between power cycling things through the apc and attempting to reconnect to them may be the key problem you are having... but I don't have any concrete rules to follow.

Here are the older notes:

Then try the disabling the VG again, cycling power to the VG through the APC gui and re-enabling the VG again:

{{{
syscmd -l 'disable_pressure_gauges()'
apcCmd off LRS2B-VacGaugeCntrlr          #  or just click on the APCgui but always check that it turned off in the APCgui
apcCmd off LRS2R-VacGaugeCntrlr          #  or just click on the APCgui but always check that it turned off in the APCgui
Current lore: restart lrs2 server    SCO, Sep12,2018
Second piece of lore:   do not do the next 3 steps --- JR,RB Sep12,2018 
apcCmd on LRS2B-VacGaugeCntrlr          #  or just click on the APCgui but always check that it turned off in the APCgui
apcCmd on LRS2R-VacGaugeCntrlr          #  or just click on the APCgui but always check that it turned off in the APCgui
syscmd -l 'enable_pressure_gauges()'
}}}

in the LRS2 log you should see

{{{
lrs2-b-vg-cntrl.as.utexas.edu:192.168.66.151:8000 connected
lrs2-r-vg-cntrl.as.utexas.edu:192.168.66.152:8000 connected
}}}

remember to wait 10 minutes before getting an accurate VG reading.

Aug9,2018 == We have been having problem communicating with the VG. This is indicated by errors when we run "ipvg off". We see complaints about being unable to reach an IP address. The fix seems to be to manual cycle VG power in the tcs gui. I.E. we turn the VG lights red then green. After that I was doing a "syscmd -l 'enable_pressure_gauges()' after the VG light was green. This seemed to put the system back into nirmal state. (SCO)


=== Hardware Changes ===

If one of the electronics day staff changes a controller a change must be made to the configuration files.    For both LRS2 and VIRUS this must be done by either Phillip, Jason or Jim so call them at home at any hour needed to get the instrument running.

=== Setting up stars to avoid a bad or dead amp ===

If one of the LRS2 amps is dead then you can try to place a point source on the other half of the IFU as to avoid that amp.   For example if you want to avoid the LU or RL amps then you will need to offset the star by +1.5 arcseconds in tracker Y from the ideal setup position or setup +5.5 pixels to larger X values on the ACAM.   Conversly if you want to avoid the RU or LL amps then you will need to offset the star by -1.5 arcseconds in tracker Y or setup -5.5 pixels to smaller X values on the ACAM.

=== Additional Help ===

Help for the complete set of CAMRA commands can be obtained by issuing a 'help()' command to the CAMRA subsystem.

{{{
#!div style="font-size: 80%"
  {{{#!bash
  syscmd -l 'help()'
  }}}
}}}