= HPF = You can find the HPF team's web site with their man at [http://psuastro.github.io/HPF/Operators-Manual/] and another useful site is their [http://psuastro.github.io/HPF/Exposure-Times/ exposure time calculator] == HPF Observing == #setup === Suggested terminals setup === #terms Easiest method to start up is to - click "RA HPF session" icon on central desktop {{{#!comment - run {{{ echo $DISPLAY }}} and copy output (something like "localhost:11.1" }}} - run "rasession" {{{#!comment - In 'HPF terminal' window (upper left) run command {{{ export DISPLAY= }}} }}} This will bring four terminal screen session Or Note that if you open new xterms you should select mcs as the host machine (not local). {{{ ---------------------------------------------- - HPF terminal - | - HET client - | ssh -X hpf@hpf | ssh -X hpf@hpf | | screen -x TIMS | ---------------------------------------------- - HPF monitor - | - HPF Scipts - | ssh -X hpf@hpf |ssh -X hpf@hpf | screen -x hpf |screen -x sc | ---------------------------------------------- }}} In the rare event that these screens sessions are not running (or you accidentally "exit"ed out of the screen session), you can create them with the following command, e.g., for the Scripts screen: {{{ screen -S scripts }}} and then connect to them as usual. === Possible script shortcut to setting up with the regular mode on the HR fiber for HPF === * Send the target from shuffle using either the IFU id 600 for regular setup or IFU id 603 for direct setups. Note if you need to use the side CB IFU use id 610. * make sure the pfip shutter is open * open a connection to htcs in an xterm or console window * in the htcs window as the structure sets down run the script {{{ ./regularhpfsetup }}} or {{{ ./directhpfsetup }}} * follow the steps as prompted * in the htcs window when the regularhpfsetup concludes run either {{{ ./guider1setup }}} or {{{ ./guider2setup }}} * tell the RA when done so they can start the exposure. === The step by step details === Setting up for an HPF target follows the following procedures: * use target_setup and either IFU 600 (normal setup) or 603 (direct setup) Note if you need to use the side CB IFU use id 610. * Example {{{target_setup 1183 E -to -ifu 600}}} * Change the acam filter to icd to /home/hpf/HPFics * Change the guider guide probe to i * Change the metrology guide probe to g * Offset to the correct position on the ACAM based on the current best positions [wiki:HetProcedures/RA/markers marker positions.] * Save an ACAM image. * open the PFIP shutter * retract ACAM and turn off the ACAM pipeline * start guiding on the correct position on the HPFACAM based on the current best positions [wiki:HetProcedures/RA/markers marker positions.] NOTE: If you have trouble getting the star to stay on the HPFACAM try increasing the integral term on the PID loop {{{syscmd -T 'HPF_ACQ_tune(kd=0,ki=0.07,kp=1)'}}} if you want to set it back to default {{{syscmd -T 'HPF_ACQ_tune(kd=0,ki=0,kp=1)'}}} to see the current values do {{{syscmd -T 'HPF_ACQ_get_tuning()'}}} * Center the guide star in the desired guider probe (gc1 or gc2) * Offset the desired guider probe by the correct amount based on the desired target science fiber: [wiki:HetProcedures/RA/hpf#heoffsets HE offsets] or [wiki:HetProcedures/RA/hpf#hroffsets HR offsets] * Do handshake from HPFACAM to the guider * Offset the fiducial by the correct amount based on the desired target science fiber: [wiki:HetProcedures/RA/hpf#heoffsets HE offsets] or [wiki:HetProcedures/RA/hpf#hroffsets HR offsets] * start HPF exposure == HPF offsets to fibers == === HPF HR mode === #hroffsets * {{{ syscmd -T 'Guider1_offset_probe (dx_ang=5,dy_ang=4)' }}} * {{{ syscmd -T 'Guider1_offset_fiducial (dx_asec=-9.6,dy_asec=-5.5,compensate="true")' }}} OR * {{{ syscmd -T 'Guider2_offset_probe (dx_ang=5,dy_ang=4)' }}} * {{{ syscmd -T 'Guider2_offset_fiducial (dx_asec=-9.6,dy_asec=-5.5,compensate="true")' }}} === Blind offsets from ACAM without using HPFACQ in the HR mode === Run shuffle with the fiber ID of 603 instead of 600. Use the HPF_HRfiber position to setup on in the ACAM. If there is uncertainty about the position then measure it at the end of a regular HPF setup to confirm the position. Do the handshake from the ACAM to the best guider choice. pull out the ACAM and start the science exposure. == Instrument Control == Here are the [wiki:HetProcedures/RA/hpf/old old instrument control commands]. Many of which are valid but we have new wrappers that are easier to use and more elegant. To get to the HPF machine: ssh -X hpf@hpf We communicate with HPF via screens to see more about screens see our [wiki:HetProcedures/RA/Screens Screens Summary Page]. To bring up the HPF monitor do a: {{{ (HPFpython) [hpf@hpfserver ~]$ screen -x hpf }}} That screen is simply for monitoring the state of the instrument. To end the screen session close the xterm/tab or do a a d. '''Do not do a d''' You will need another ssh window to make commands to the instrument {{{ssh -X hpf@hpf }}}. In this window you will control the lamps, the comparison lamp shutters, exposure times etc... grab the scripts screen session {{{screen -x scripts}}} session so that any failures can be diagnosed by the HPF team. Data is stored in {{{/HPFData/LinuxGigE/YYYYMMDD/hpf}}} from other machines, e.g. mcs, or {{{/home/hpf/HPFData/LinuxGigE/YYYYMMDD/hpf}}} on the hpf machine. Scripts for changing the instrument settings are located in /home/hpf/Scripts/InstConfig === Controlling the Laser Frequency Comb (lfc) === Some targets, usually the brighter ones and the precision radial velocity targets, will want to use the lfc to get the best possible radial velocities. The lfc remains on at all times but to include it in the observation you need to open its internal shutter. To turn it off and on use the following commands: {{{ lfc_open }}} {{{lfc_close }}} In some cases, the laser comb may be down and we must use the following: {{{ etalon_open }}} {{{ etalon_close }}} Note that each of these command sets are executed in the Scripts window. The open command will usually take a few seconds to complte and you will see a variety of messages coming to the window. After PartA cals are done, I turn on the etalon with: {{{ ./hpf_TurnOn.sh Etalon }}} That takes 10 minutes to warm up. Then, throughout the nights, I try to take the etalon cals at least once an hour: {{{ ./hpf_takeEtalonHRCalExposure.sh 3 }}} Doing 3 of those takes about 9 minutes. You can run: {{{ ./hpf_takeEtalonHRCalExposure.sh 1 }}} to run a single ramp (takes 3 minutes). For targets which request LFC, I think our approach right now is to: --keep cal_closed --run etalon cals before observing the target (while slewing/setting up) --run etalon cals after the target At the end of the night turn off the Etalon {{{ ./hpf_TurnOff.sh Etalon }}} === Engineering observing === The command to execute any basic observation is hpframp targetname #reads_per_ramp #ramps [obs#]. Each read is 10.5 seconds so a 30 read observation would be 315 seconds. There is a 2 read reset at the beginning of the exposure that adds to the overhead. You can include a optional observing number, but if the software has not been restarted and you do not include this observing number it will use the next observing number. If you restart the software the count starts again at 1 and the next target will be included in directory 0001 even if there is data in that directory, however, due to the time stamp no data will be overwritten. Example: {{{ hpframp AD_Leo 30 2 }}} You can set the type to Eng so that they are not science exposures and sent to the TACC by doing a {{{ tims detect pyhxrg:SetObstype:Eng }}} === Science observing === #obs run all of our science frames and cals in the {{{screen -x scripts}}} session so that any failures can be diagnosed by the HPF team. The command to execute any basic observation is hpfseconds targetname exposuretime #exposures [obs#]. If the target name has a space we should use double quotes around the name, e.g. "please do not use spaces in target names" Wtail -f OCDlog_20181210_C.txt | grep DEXhere the exposure time is in seconds. You can include a optional observing number, but if the software has not been restarted and you do not include this observing number it will use the next observing number. If you restart the software the count starts again at 1 and the next target will be included in directory 0001 even if there is data in that directory, however, due to the time stamp no data will be overwritten. Examples: {{{hpfseconds GJ_3470 315 5}}} - 5 exposures of 315 seconds each. {{{hpfseconds GJ_3470 315 5 100}}} - 5 exposures of 315 seconds each to be saved in directory 0100. You can look at any spectrum using showhpfspec YYYYMMDD obs#. If you leave off any date and observation number it will show the last exposure. Examples: {{{showhpfspec}}} {{{showhpfspec 20180428 2}}} === Aborting normal science exposures === To stop after the current exposure (before the next ramp/exposure starts) you will need another hpf xterm and in there do a {{{sc}}} If you need to stop one of the observing scripts, like PartA, then try a cntl-C. This will stop the script and at the end of the current exposure (which you can monitor from the HPF monitor screen) you should be able to resume working with HPF. === Standard stars === For telluric standards you should not observe anything brighter than V=4.5 (our lists have been culled to remove these and emission line objects). If you find that any telluric is in a visual binary with a brighter star please please feel free to remove that from the list and let astronomer@het.as.utexas.edu know so that we can edit the online list. For exposure times please use: * 315 sec exposure (30 reads) for anything fainter than V = 5.8 * 105 sec exposure (10 reads) for anything between 4.5 and 5.8 in V Be sure to edit the night report to change the Data Type to "telluric" **NB** Telluric standards must be taken with LFC closed == Closing the HPF terminal windows before an mcs reboot == #HPFmcsreboot === Starting the HPF windows (method 1) === * From a gnome-terminal windo hit the File menu and select "Open Terminal" then "HPF". * At the prompt enter "$ ssh -X hpf". {{{#!comment * run {{{ echo $DISPLAY }}} and copy output (something like "localhost:11.1") }}} * At tthe prompt enter "$ rasession". {{{#!comment * In 'HPF terminal' window (upper left) run command {{{ export DISPLAY= }}} }}} === Starting the HPF windows (method 2) === * Click the icon named "RA HPF session" on the upper right of the middle RA monitor. {{{#!comment * run {{{ echo $DISPLAY }}} and copy output (something like "localhost:11.1") }}} * In the window that comes up (which is already in full screen mode), ender "rasession". * The four hpf windows appear {{{#!comment * In 'HPF terminal' window (upper left) run command {{{ export DISPLAY= }}} }}} === Stopping the terminal (essential before mcs reboot) === * Click the small X in the upper-right corner of the terminal window. Or Just press Alt-F4 to kill terminal window == Stopping and starting the client == #Startup === restarting the the HPF client === '''Note that the HPF client is restarted automatically every day 16:45 UT just after the LN2 refill. The act of restarting the HPF makes a small change to the instrument temperature which impacts RV stability.''' {{{ restart_detectorclient }}} is called daily as part of the crontab at 16:45 UT to restart the HPF client daily. It can also be called on demand. There are two new executable scripts: 1) start_hpfdetectorInfloop.sh - this is running in the 'hpf' screen session, and takes care of automatically restarting the client after it is shutdown 2) restart_detectorclient - can be called from any hpf terminal (but preferably the 'scripts' session. This will trigger a shutdown sequence. After the shutdown, 'start_hpfdetectorInfloop.sh' will automatically take care of bringing things back up. Check that the observation number is correct with {{{ pyhxrg:GetObsnum }}} if it is not then use the following command to set it. {{{tims Detect pyhxrg:SetObsnum:nn}}} where nn - number from previous GetObsnum command === How to stop the client === If you need to completely power off the detector and prevent a restart, there are two options: * tims detect pyhxrg:GetObsnum * touch ~/stopdetectorloop * tims detect pyhxrg:PowerDown * tims detect shutdown This will create a file in $HOME called stopdetectorloop. If that file exists, the infinite loop will not continue following the normal powerdown/shutdown sequence. * '''There is a 10 s pause between completing the power down and auto- restarting. You can press CTRL+c in the 'hpf' screen session during that time to abort the script.''' You should see something like {{{ Configured HxRG Sending Cli15380: Detect Cli15380 PowerDown:OK Resetting Asic for JADE1 ... Success Powering Down Analog Supply Voltages: VDDA ... Success Powering Down Digital Supply Voltages: VDD3p3 ... Success VDD2p5 ... Success VDDIO ... Success VSSIO ... Success Powering Down Voltage Regulators: VDDIO ... Success VDD2p5 ... Success VDD2p5 ... Success Powering Down Analog Main: 5V ... Success De-Initializing Hal Powered Down. }}} {{{ Shutting down this client. Stopping scheduled loops: ...lakeshore336 Shutting down subclients ...pyhxrg ......Failed ...lakeshore336 ......Success Sending Cli16420: Detect Cli1642 * tims detect pyhxrg:SetObstype:Sci0 message:Detect_shuttingdown Shutting down... Connection lost to: lakeshore336 Reason : [Failure instance: Traceback (failure with no frames): : Connection was closed cleanly. ] Closing connection to server Client stopped. }}} When you restart the system you should check to see if it has the correct observation number (assuming you have not rolled through 0 UT). === Restarting communications with the HET === #RestartHetClient 1) Find the 'HET Client' window in the 'TIMS' Screen session by doing a CTRL-a n (multiple times, until the window title says 'HET Client'). If a TIMS screen session is not open open one in an '''xterm''' with {{{ (HPFpython) [hpf@hpfserver ~]$ screen -x TIMS }}} 2) In the HPF Scripts window or a new hpf xterm execute : {{{ tims het shutdown }}} . This will stop the dictionary python stuff in the 'HET Client' window. 3) If you don't have a command prompt in the HET Client window, CTRL-c the client. If you do end up doing this you might want to run another tims het shutdown just to make sure everything is down. 4) Run the command 'timshet' in the HET Client screen window {{{ timshet }}} Note that this is the only TIMS client startup command that will not run with the normal startup syntax (e.g. python -m TIMS.clients.tims_het). This is because the library environment needs to be changed to access the het system libraries. We only want to do this for the HET client, so I've wrapped the normal startup command in a bash script that lives in ~/bin/ . == HPF twilight sky flats (evening) == These are not required nightly, but desired when possible. Only take HPF twilights '''if the sky is clear'''. Brightly lit clouds can cause HPF to saturate. Run, on MCS, 2 minutes after sunset: {{{ htwi }}} Can view current incoming data with showhpfspec (after 3rd frame is captured) Can also view fits files in: ~/HPFData/LinuxGigE/YYYYMMDD/hpf/OBSNUM/fits/ (note there is a 11k bias level) == HPF Darks and checking for saturation == If you need to take a dark exposure (i.e., to check for saturation), this is the procedure (in the {{{CalScripts}}} folder). First, make sure PFIP shutter is closed. Then, run: {{{ ./hpf_setup_ToTakeDark.sh }}} {{{ ./hpf_expose.sh Dark 30 1 & }}} == HPF cals in the evening and morning == #cals run all of our science frames and cals in the {{{screen -x scripts}}} session so that any failures can be diagnosed by the HPF team. ATTENTION. If you need to pause cals use: - {{{pausehpf}}} ({{{playhpf}}} to continue) if you want to pause the script before the next hpf exposure stars - {{{pausecal}}} ({{{playcal}}} to continue) is the command to pause the cals before it stars a next set of observations before using the FCU head. Historically this command was made to pause the cals temporally for anybody on side to switch on the lights in the dome, or use fcu head for something else, etc. It will not stop any internal hpf cals. ==== Evening ==== The evening 'A' calibration sequence includes the LFC and Etalon frames (and flats) that are necessary to start the nightly drift calibration. The 'C' calibrations contain the UNe lamps, which are only present as an emergency backup source in case the LFC is offline. Taking the 'A' sequence is critical; skipping the 'C' sequence if you are crunched for time is OK, although that sequence has been timed to fit within a typical stack time. * The RA performing Ops in the afternoon will typically start the eveening HPS cal set. This will take about 44m to complete. {{{ [hpf@hpfserver CalScripts] cd /home/hpf/Scripts/InstConfig/CalScripts [hpf@hpfserver CalScripts] ./hpf_evening_cals_PartA.sh }}} If you don't have the 44m minutes for the full PartA cals just run the [[br]] {{{ ./matt_shortflats.sh }}} 22 minutes[[br]] or [[br]] {{{ ./matt_wave.sh }}} 14 minutes[[br]] depending no how long you have before sunset. '''The most critical cals are the wavelength (wave) cals.''' * The night RA will check that the job is complete by confirming that the command prompt has returned in the HPF command window. If the job is still in execution, we'll see a sequence of integers being written to this window. Also, if the script is still executing, we'll see update message appearing in the HPF monitor window. * The night RA will start the last set of HPF cals during stacking (or when possible). This last step, referred to as the "C" script will take about 43m. Note that we can take VIRUS or LRS2 cals during this (internal) part C calibration run. {{{ [hpf@hpfserver CalScripts] cd /home/hpf/Scripts/InstConfig/CalScripts [hpf@hpfserver CalScripts] ./hpf_evening_cals_PartC.sh }}} * Combining A+C cals takes 1h 27m, and could be initiated as part of the afternoon OPS if dome stays closed. * *NOTE on order of A,C cals*: * Part C cals must be run AFTER Part A for lamp warmup/shutdown reasons. When Part A is run, it leaves the UNe lamp on. This lamp is needed by Part C, and Part C turns it off when it finishes. However, if Part C is not run after Part A, you must turn off the lamp manually with: {{{ [hpf@hpfserver CalScripts] /home/hpf/Scripts/InstConfig/CalScripts/hpf_TurnOff.sh UNeS }}} * Similarly, if you are running Part C without first running Part A, you must turn on the UNe lamp via: {{{ [hpf@hpfserver CalScripts] /home/hpf/Scripts/InstConfig/CalScripts/hpf_TurnOn.sh UNeS }}} We have added an additional script (hpf_takeLFCHRCalExposure.sh) that will take a LFC exposure through the calibration fiber (~2 minutes to complete). This should not interfere with any other telescope operations, as it is entirely internal to HPF. If you find yourself in a situation similar to last night, you can run this stand alone script a few times while a different primary operation is happening (e.g. LRS or VIRUS exposure, slew, or other telescope setup). That will at least provide a few LFC reference points to tie the drift calculation to. The script accepts an optional argument - N - where N is the number of ramps you wish to take. e.g. {{{ ./hpf_takeLFCHRCalExposure.sh 3 }}} ==== Morning ==== * The night RA will start a morning calibration for HPF. This will take about 50m to complete the portion requiring the FCUhead and a dark dome. An additional 1h10m with the pfip shutter closed is required to complete the full calibration script run. {{{ [hpf@hpfserver CalScripts] cd /home/hpf/Scripts/InstConfig/CalScripts [hpf@hpfserver CalScripts] ./hpf_morning_cals.sh }}} ==== Stopping Cals ==== If you need to stop the cals then you just do a cntrC in the scripts window which will stop the script but will '''NOT''' stop the current exposure. Watch the HPF monitor screen to see when the instrument stops the exposure and goes idle. == Trouble shooting and restarting == #troubleshooting ==== Arrowing up to see what happened ==== If you missed something that has scrolled off the top of the screen and you want to see what it was you can do a a followed by an to get into "copy" mode. Then use the arrows to move the cursor up to get to the message of interest. To leaves copy mode hit two in a row. To see the full set of screens commands please see our [wiki:HetProcedures/RA/Screens Screens Summary Page]. ==== Observing if the LFC is down ==== If the LFC is down but was requested, you can use the Etalon as a calibrator. Instead of running the LFC setup scripts, run the following script: hpf_setup_ToTakeHRCalFib.sh Etalon This will turn on the etalon and open the shutter To turn off the etalon, run the following two scripts: hpf_TurnOff.sh Etalon hpf_CloseCalShutter.sh HRCal Etalon Joe has created two new scripts for the etalon: etalon_open etalon_close These can be used just as you would have used the equivalent LFC scripts. ==== Fits Key words ==== See [wiki:HetProcedures/RA/hpf#RestartHetClient Restarting communications with the HET] {{{ #!comment If during the night you find that these keywords are not being captured, you can restart the client using the following commands listed below: 1) From any terminal: (HPFpython) [hpf@hpfserver ~]$ tims het shutdown (As I type that, it occurs to me that the command has a very dire sound to it... I assure you it will not shutdown the het... just HPF's connection to it.) 2) Open the 'TIMS' screen session (HPFpython) [hpf@hpfserver ~]$ screen -x TIMS 3) Switch to the 'HET Client' window (Note this is different that the 'HET Relay' window!) CTRL-a n (multiple times, until window title says 'HET Client') 4) You should see a bunch of dictionary output. If you don't have a command prompt, CTRL-c the client 5) Run the command 'timshet' in the HET Client screen window (HPFpython) (HPFpython) [hpf@hpfserver HPFics]$ timshet Note that this is the only TIMS client startup command that will not run with the normal startup syntax (e.g. python -m TIMS.clients.tims_het). This is because the library environment needs to be changed to access the het system libraries. We only want to do this for the HET client, so I've wrapped the normal startup command in a bash script that lives in ~/bin/ . }}} === Restart Client when it stop working === If see 'Error' in hpf session of client is not operation normally you can try to restart client If you get the '''QSocketNotifier:''' in the detector client window you can try restarting it but if it comes back then the instrument is down and you should call Chad. Here is restarting procedure: HPF [wiki:HetProcedures/RA/hpf#Startup Starting and Stopping the Client ] === Problem with LFC === If part A script hang on message: WARNING: Setting LFC flux to High .. means absence of communication to LFC computer. It possible to abort script with Ctrl-C and report to Chand, Joe and Connor Until problem fixed it is better to assume that LFC is down and use Etalon Use the following replacement scripts * {{{ hpf_evening_cals_PartA_withoutLFC.sh }}} '''NOTE this takes 53 minutes to complete which is longer than the old Part A''' * {{{ ./hpf_evening_cals_PartC.sh }}} This one has no change because it does not interact with the LFC. * {{{ hpf_evening_cals_PartAC_withoutLFC.sh }}} This is just like the usual PartAC but calling the LFC-less partA. * {{{ hpf_morning_cals_withoutLFC.sh }}} '''NOTE this takes 26 minutes of dark dome to complete and 1 hour 15 minutes pfip restriction''' The etallon is brighter than the LFC so if you take an exposure longer than 150 seconds with the etallon open you will likely saturate the detector. Thus, do not use the etallon during an observation unless the exposures are less than 150 seconds. Instead, taken an single etallon before and after each HPF observation. === Fixing the calibration connection === If the etalon turn off command hangs or any of the cal lamps try the following: {{{ tims Calib shutdown }}} in any hpf window with a prompt In the TIMS HET window do a '''cntrl-a n''' five times to move to screen 2 which is the TIMS Calib screen. Note that we must hit cntl-a then hit the n (for next srccen). This full sequence is repeated to go to the next screen. The screen we usually operate in has the heading name "Screen 4: HPF HET Client". To fix the probelm of Q-data missing from the HPF headers (for example) we need the screen that is labeled "Screen 3: HPF TCS Relay". The screen we start from is named "Screen 4: HPF HET Client". The screen we want to end up at is It should have a prompt. If it does not you may need to do a few cntrl-c to kill any hanging processes. In the TIMS Calib screen you should do an up arrow and see the {{{ python -m TIMS.clients.tims_calib }}} command which restarted the Calib client. Wait a few minutes for it to finish. Take a quick lfc image and perhaps a quick sky exposure to make sure all of the shutters and baffles are working. === Fixing the loss of tcs info in hpf headers === {{{ From Chad Bender (Feb17,2019): Here is what I did to restart both the TCS relay and the TCS Client. In the TIMS Screen session: 1) Change to HPF TCS Relay window (currently #3, but not guaranteed to always be so) 2) CTRL-C the script running there (this is a script, not a TIMS client). You will need to CTRL-C twice to actually kill it. 3) Restart the script (up arrow or ./bashwrapperfor_HETTCSRebroadcast.sh) 4) Restart the TIMS HET Client in the normal manner For this last step, we use: tims het shutdown # in the Scripts window timshet # in the HPF HET Client window (usually upper-right screen) }}} === Listening to the fiber scrambler === As of Feb 2019, you can use the "RA" knob on the audio box (under the UT clock) to control the volume of the audio feed from HPF calibration enclosure which plays on the speaker under the RA desk. You can also connect to this audio feed with VLC media player on MCS via: {{{ cvlc rtp://@:4444 --sout-rtp-caching 40 }}} (drop the leading "c" if you want a UI)