Overview of the basic night time procedures

Afternoon checkout

suggested order of OPS procedure.

• Check VIRUS/LRS2 temperatures and pressures (all in the temperature list now). If pressures are not given (i.e., appear as "------"), the VG needs to be enabled and one needs to wait 10 minutes before trusting the pressure readout.
• turn off ION PUMP, VG and VIRUS V309_IONPump using the command ipvg off -V confirm IP is off on APC GUI. The VG is only disabled not turned off with this command. If you need to do this with low level commands you can turn off the IP on the APC directly but Important!!!! Leave LRS2 VacuumGauge? on on APC GUI. To disable or enable VG use commands

  syscmd -l 'disable_pressure_gauges()'  # should be run during OPS after pressure reading
  syscmd -l 'enable_pressure_gauges()'   # should be run RA in the morning
  

• check instruments temperatures
• check IFU's percentage power (Left and Right Pwr %). Adjust any temperatures as needed.
• take lrs2 and virus test cals
• restart HPF HET client and make sure the observation numbering has not been reset.
• do test move of the guide probes (home4 or home5)
• do the acam stability test
• get acam biases test
• add the LRS2 and virus uptime getUptime --lrs2 --virus
• Turn off dome lights
• take HPF cals and any long cals that can be done

When servers are DOWN

• On TO Launcher on TO console > Operations > TCS Servers > All Servers > Restart
• A window will pop up showing each server being started/restarted

Detailed list

Only exposure numbers from 9900-9999 are to be used for afternoon OPS work

Check the temperatures and voltages of the instruments

• LRS2 temperatures:
  • For LRS2, use the RA launcher application (Operations drop-down menu) and select Display LRS2 Temperatures
    • CCD temps (L,R for Red, & Blue) should be around -115.0
    • Cryo temps should be around -169.0
    • Cryo Pressures should be displaying
    • the temp check window remains open for 30 seconds
    • check to make sure that the % power are all between 5 and 90%.
  • Sometimes the RAlauncher approach does not work, so you might try

       $ getSpectTemperatures --virus
       $ getSpectTemperatures --lrs2
    

  • Frequently temps will not report for LRS2 and it is useful to restart the server:

        [astronomer@mcs ~]$ ssh -X sco@lrs2
        [sco@lrs2 ~]$ sudo service ldas restart
    

    Wait for several minutes until the green LRS2 light appears in the TCS gui.
    
• VIRUS temperatures:
  • For VIRUS, use the RA launcher application (Operations drop-down menu) and select Display VIRUS Temperatures
    • NOTE: If you just restarted VIRUS within the last hour the temperatures and power might not be stable
    • check to make sure all power percentages (last two columns labeled Left Pwr % and Right Pwr %) are between 5 and 90%.
  • If below 5% AND the Set Point is < -95 then follow the procedure below. If above 90% AND the Set Point is > -115 then follow the procedure below.
    • Bring up the TCS gui and go to the Virus Environment tab
    • Within the Virus Environment tab go to the Warm Up/Cool? Down tab
    • Enter the Virus spectrograph ID number into the Spec entry form. Example for V320 enter 320.
    • Hit the Update Temps button and then look at the Set Point values on the listed under Left CCD and Right CCD
    • Enter BOTH your new and any old target temperatures under Target L and Target R. NOTE: Both of these need to be filled out as both are set when you hit Execute.
      • If the power is below 5% then raise the temperature by 5 degrees. If the power is above 90% then lower the temperature by 5 degrees. For example if the Left and Right set points are -110 and -105 and the Left and Right Pwr % are 21.12 and 4.99 then you will be entering the Target set points as -110 and -100 (the left did not change while the right is now 5 degrees warmer).
    • Hit Execute when BOTH fields are entered.
    • Look at the Left CCD Set Point and Right CCD Set Point on the Gui to see that they changed.
    • NOTE: It may take several tens of minutes for the actual Temp to get to the Set Point and during that period the Pwr % may go to 100% or 0% depending upon which way the set point moved.

Check LRS2 vacuum pressure

• In the LRS2 Temperatures display, the VG pressures are listed. If they say "------" that means that VG need to be enabled and one needs to wait 10 minutes before trusting the pressure readout.

• If the servers are not up you will need to ask the TO or day staff to do this through the pull down menu of the TOlauncher on jove.

Turn off the LRS2 ion pump and Vacuum Gauge

• In an mcs window do a ipvg off -V

Take test exposures:

• Take bias imgags first to check if LRS2 and VIRUS are fine

cal lrs2 -B -b 1 -pb -o 9900 -O 9900

• The following check of VIRUS should take about 7 minutes to complete:

cal virus -conf /opt/het/hetdex/etc/configfiles/vl_cals_ops.json -l ldls_long cd-a hg -L 1 -B -o 9901

• The following check of LRS2 should take about 10 minutes minutes to complete:

cal lrs2 -conf /opt/het/hetdex/etc/configfiles/vl_cals_ops.json -l qth fear -L 1 -B -d 1 -de 300 -o 9901

• Recall that you can use the same log name for multiple runs (the results will be append each time you run cal.py)
• Examples of images: Sample cal images for VIRUS
• Run vhc on the VIRUS images: see How to run vhc.
• Make a summary of the VHC output for the report by using viruserr
• Verify that LRS2 and VIRUS exposures have appropriate header keywords with telemetry from TCS

Make sure the /mnt/camra_ramdisk disks on lrs2 and virus are empty (ssh to 'lrs2' and 'vdas'). If they are not that may indicate a problem with data transfer.

Make sure that the /var partition on MCS is not full. df -h | grep var produces: /dev/sda2 7.8G 4.4G 3.0G 60% /var

Make sure that MCS is playing sounds at the completion of exposures/scripts. You can generate a test sound with: play -q /opt/het/hetdex/etc/sounds/alert-1.ogg

If sounds are not working, try as many of these as you can, in this order:

• run pulseaudio -k
• run rm /tmp/.esd-*/socket and then pulseaudio -D
• run (as root): ps ax | grep pulse , kill <pulsaudio PID> , and /usr/bin/pulseaudio --start --log-target=syslog
• logout from astronomer and login again
• reboot mcs

Check that the HPF numbering has not been reset (if before 0 UT):

• in the HPF scripts window run tims detect pyhxrg:GetObsnum

Restart HPF HET client:

• find the 'HET Client' window of the TIMS screen (currently screen 7).
• shutdown the het communication client: tims het shutdown from the HPF Scripts window or a free xterm on hpf.
• in the 'HET Client' window of the TIMS screen restart the communication client with the command timshet

Verify that HPF exposures have header keywords with appropriate telemetry from TCS.

Test guide probe motions:

• run shoprobes for current positions
• Then run:

  [astronomer@mcs ~]$ home6  (or home5, home4, etc...)
  

• Watch probe display ("Probe Assembly" window) on tcs gui to insure that probes move

Take a set of 5 ACM bias images:

• In the run below we use a script to take 5 acm bias images and this will take about 75 seconds to complete. The script needs to know the UT date for which the data are being collected. This is the date string gnerated by the 'cwd' script on mcs (as astronomer). Enter the UT date string when you are queried by the script at run time.

  [astronomer@mcs ~]$ cd ~astronomer/acm_bias
  [astronomer@mcs ~]$ acm_bias_run 5 
  1375.5416 0.0451 29.3489 25 20190220T182024.6_acm_sci
  1375.5341 0.0450 29.3068 25 20190220T182039.6_acm_sci
  1375.5309 0.0450 29.3163 25 20190220T182048.8_acm_sci
  1375.6678 0.0451 29.3683 25 20190220T182058.0_acm_sci
  1375.6355 0.0454 29.5571 25 20190220T182107.2_acm_sci
  Number in first subset file = 5
  

• Some other output is generated, but is not very important for the user. In the case above we see some simple statistics generated for each acm bias image. The first number is the mean bias level in the raw frames. This is usually in the range 1374 < Mean_BIAS < 1386 (adu units). If the values depart from this range by (for instance) tens of adu, then it s likely that some camera (like PV) has been left on and is creating scattered light in the pfip.
• You can read more details about Calibrations for the acm
• I made another link A practice link for attachments

Test the ACAM position:

• Make sure the RA and TO are not doing anything with the PFIP or PAS
• ask the TO to put the tracker at center
• run command acq_camera_mirror_test <tracker temp> (e.g., acq_camera_mirror_test 23.5 getting tracker temp from HET weather GUI)
  
  • or use the old version in the astronomer/bin directory: acqm.py <tracker temp> (e.g., acqm.py 25.0)
    

or try the following manual steps

• execute the script /home/jove/guider/chevo/shell/acamtest/acqmirror (be sure to not let this span the UT date roll over)
• do a ls -t -1 /hetdata/data/`cwd`/acm/20* | head -10 > /home/mcs/astronomer/iraf/ilist
• start iraf
• run xregister on the ilist with /hetdata/data/20180327/acm/20180327T024656.8_acm_sci.fits as the reference with the region set to [160:360,333:600]
• calculate average and standard deviations for both X and Y

• report in the RA OPS report along with the Tracker temperature:

  For acam mirror test (Tracker Temp = 16.9 C):
                       X(pixels)       Y(pixels)
  Mean shift            3.173             3.193
  StanDev               0.047             0.024
  

Get the LRS2 and virus uptimes via:

• getUptime --lrs2 --virus

Check the QueueTool? tab on the nossy RA interface for any new allocations or target submissions/modifications. Verify and accept these.

Check for any high priority targets for that night by running "priscript.sh". It may be best to first 1) accept all new targets, 2) run "shuffle_daily" on vdas, and 3) use the "Daily Calcs" button in HTOPX's Tools menu to compute all synoptic constraints for today.

Make sure the basement lights are off and check to see if the dome lights can be turned off.

If you can make the dome dark:

• Start a set of HPF Part A+C cals (if time. As of 13 Sep 2018 PartA takes 44m and PartC takes 43m)

e.g., /home/hpf/Scripts/InstConfig/CalScripts/hpf_evening_cals_PartAC.sh on HPF Scripts window

• Record the start and estimated stop time for the Part A+C cals in the report.
• Verify that you hear the HPF fiber scrambler after Part A darks have begun (if no sounds, call George, Chad, etc)

Problems you might encounter

• get_hardware_status does not return values: restart camra server
• camra server does not restart: See problem solutions for camra

If there will be time in dark dome conditions (or when hpf PartC is running) perform LRS2+VIRUS cals:

• If this might span 0 UT OR if you want to run both LRS2 and VIRUS lamp cals use sir script, e.g. /home/mcs/astronomer/sir/evcals.sh <-l> <-V> [-b|-pb] [-v]
  • Use the "-h" flag to get more details
  • /home/mcs/astronomer/sir/evcals.sh -l -v takes LRS2 lamp cals in 44 minutes plus any wait time around 0 UT.
  • /home/mcs/astronomer/sir/evcals.sh -V -v take VIRUS lamp cals in 38 minutes plus any wait time around 0 UT.
  • /home/mcs/astronomer/sir/evcals.sh -pb -v takes combined bias frames in 14 minutes plus any wait time around 0 UT.
  • /home/mcs/astronomer/sir/evcals.sh -l -V -pb -v takes all LRS2 and VIRUS cals in 1 hour 35 minutes plus any wait time around 0 UT.
• If the cal of interest will not span 0 UT OR is not both LRS2 and VIRUS lamp cals then use normal cal script, e.g. cal lrs2 -l hg cd-a fear -c qth ldls_long -B , cal -pb -B or cal virus -l hg cd-a -c ldls_long -b 0 -d 1 -de 360 .
• To run the parallel biases at a certain time (i.e., if you know when PartA finishes and PartC starts), you can use:

at 23:40

/home/mcs/astronomer/sir/evcals.sh -pb -v

[ctrl-D]

OR

at 23:20 (note how this cal of interest would complete before the 0 UT roll-over)

cal lrs2 -l hg cd-a fear -c qth ldls_long -B

[ctrl-D]

Start of the Night

• Start an RA log named YYYYMMDD_RAlog.txt (can do it in /data1/mcs/guider/YYYYMMDD/ )

• Check the temperatures of the instruments (via RA Launcher-->Operations)

• Check on status of the LRS2 ion pump and VG (in TCS)

• Check e-mails to see if PV or other engineering sky flats are required (file unneeded; flag necessary)

• Check latest commissioning plan located here or Science plan located here

• Use chksys to see if any defunct processes such as the APC or Gui exist. Kill those processes.

• Start Monitors for instruments and TCS see LRS2 and VIRUS procedures.

• Walk around dome area to make sure lights and mothenators are off.

• NEW PROCEDURE FOR TWILIGHTS
  • Take twilight spectra (start GC2 with "VIEW ENXTENSION" = raw for B filter and exp 0.1s
  • IF no clouds start lsky.sh when GC2 flux is 36k
  • ELSE if there are clouds then start lsky.sh when GC2 flux is 35k
  • Start VIRUS script vsky.sh when GC2 flux is 6.6k

• IF possible (NOT required nightly, unless requested), and only if the sky is clear (bright clouds can saturate HPF), take HPF twilight spectra:
  • on mcs execute htwi (starts twilight exp immediately) or htwi w (wait for sunset + 2 min)
  • OR use the old manual procedure...
  • on the hpf machine execute tims detect pyhxrg:SetObstype:Cal
  • open the PFIP shutter, make sure ACAM and FCU head are retracted
  • AT sunset, run hpframp twilight 28 1
  • close the PFIP shutter
  • tims detect pyhxrg:SetObstype:Sci
  • this should produce nice twilight flats for HPF, but may eliminate your chance of getting LRS2 sky flats that night

• Take twilight spectra (start GC2 and check flux level - ~22k LRS2 and 4.7k virus for B filter and exp 0.1s); NOTE: 30k for LRS2 and 5.5k for VIRUS seemed to work better in the summer. If you end up using GC1 on filter B then you should use ~36k for LRS2 and ~6.9k for VIRUS.
  • First any imaging cameras
  • LRS2 spectra
  • VIRUS spectra
    

• Evening twilights are lsky.sh and vsky.sh
• Morning cals. ~3k virus vsky-morning.sh and ~15k lrs2 lsky-morning.sh

For LRS2:
vlexp -i lrs2 -texp 15 -t twi -pobj skyflat
or
iexp lrs2 -skyflat

For VIRUS:
vlexp -i virus -texp 15 -t twi -pobj skyflat
OR
specialtwilight

• While stacking take the LRS2 and VIRUS biases.

• Start DIMM if weather permits. For directions see DIMM wiki page

End of the Night

Data should transfer itself to the TACC so no action from the RA should be required.

End of the night e-mails should be done automatically so no action from the RA should be required.

Take end of night calibrations:

• run eon_tasks script to get
  • end-of-night VIRUS bias. Use vds and save image.
  • extra VIRUS cals for Gary
  • Turn on LRS2 ion pump and VG
  • displays LRS2 temperatures and VIRUS temperatures
• Take any cals needed for science targets observed.
• Take HPF end of night cals.

Park the instruments

• Make sure the LRS2 and VIRUS servers, pivot and monitors are running
• Use chksys to see if any defunct processes such as the APC or Gui exist. Kill those processes.

Move the RA log into the Guider nightly working directory /data1/mcs/guider/YYYYMMDD/

Shut down the DIMM software and close the DIMM. See DIMM wiki page