Het Tracker Incident 14 Jan 2005
24 January 2005
James R. Fowler
George Damm
On the night of 14 January 2005 CST, while the Operations Engineer was preparing the telescope for a science night, the tracker assembly lost the correct values of the encoder positions and appears to have hit the negative X hard stops at the slew speed. The tracker may also have been forced against the negative hard stops by the track motors. These actions appear to have forced the lower X tracker brake disc against the brake pad support. The increased friction on the brake disc prevented the lower X track motor from moving the tracker out of a skew situation.
A physical and electrical inspection of the lower X assembly revealed that the encoders were lost, that the tracker was in a skew situation (approximately 30 mm), and that the lower X track motor could not move the tracker. Vicki Riley, Sergey Rostopchin, George Damm, Edmundo Balderrama, Robert Poenisch, Bob Calder and Jim Fowler assisted in troubleshooting the problem. Jim Fowler reviewed all software logs while George Damm explored the physical aspects of the tracker system. This information was used by Jim Fowler and George Damm to prepare this report.
The investigation revealed that the most likely cause of the incident was a lack of attention to the position of the tracker caused by inexperience and a lack of adequate limit safeties. An incorrect skew recovery procedure also lead to an additional night of lost time. The following scenario is considered to be the most likely sequence of events.
During system start up prior to the start of night time operations the Operations Engineer incorrectly initialized the tracker encoders twice. This resulted in the tracker being some 2000 mm negative in X of the positions reported to the operator. During this time the tracker was sent to the center of its travel using the slew motors. Because the encoders were incorrect this caused the tracker assembly to hit the hard stops at the negative end of the X travel. The tracker may also have been pushed against the hard stops with the track motors as well. These motor provide a high torque at low speeds. The high speed impact and/or the force of the track motors against the hard stop forced the track motor brake disc against the brake pad support assembly. The increased friction on the brake disc then prevented the lower X motor from begin able to move the lower X axis of the tracker.
The following actions, listed in order of priority, are recommended to prevent a recurrence of this incident and to provide better data collection in the event of future incidents.
Change the software limits Ix113/Ix114 in the Pmac to be ±1870 mm or move the limit switches.
Add additional logging to the Tss program. All commands should be logged as well as any error conditions, e.g. limit switch activation.
Install additional limit switches at ±1700 on both X axes and the Y axis to provide the correct limit indication to the slew motor.
Add additional logging of engineering information to the Tss program. Review data to include in such a log, e.g. commanded positions, actual positions. Such logs will provide improved information about the software’s knowledge of the tracker position and will provide information required to track motor performance.
Modify Tcs Go Next button to do nothing if no valid object exists.
Revise Skew Recovery Procedure to properly start up motors.
Provide additional training to TOs and engineering staff about tracker operations.
Review requirements for implementing a procedure to do real time correction of the position recorded by the encoders by monitoring the position of the index marks. Determine the amount of work required to implement such a system and if such a system would provide improved performance and safety.
Devise procedure and installation requirements for installing hard limit switches between the current limit switches and the hard stop.
On the night of 14 January 2005 CST, while the Operations Engineer was preparing the telescope for a science night, the tracker assembly lost the correct values of the encoder positions and appears to have hit the negative X hard stops at the slew speed. The tracker may also have been forced against the negative hard stops by the track motors. These actions appear to have forced the lower X tracker brake disc against the brake pad support. The increased friction on the brake disc prevented the lower X track motor from moving the tracker out of a skew situation.
A physical and electrical inspection of the lower X assembly revealed that the encoders were lost, that the tracker was in a skew situation (approximately 30 mm), and that the lower X track motor could not move the tracker. Vicki Riley, Sergey Rostopchin, George Damm, Edmundo Balderrama, Robert Poensich, Bob Calder and Jim Fowler assisted in troubleshooting the problem. A review of the software logs revealed the most likely cause of the incident was a lack of attention to the position of the tracker caused by inexperience. An incorrect skew recovery procedure also lead to an additional night of lost time.
This report describes the sequence of events that occurred during the incident with commentary about the events for those unfamiliar with tracker operation. It then describes the Operations Engineering procedure followed by a simple description of tracker operation during an X motion. Finally the most likely cause of the incident is described and a list of recommendations provided to prevent a similar incident in the future.
Data logs and information about the state and history of the Tracker system as well as the Telescope Control System (Tcs) are available in a number of locations. The Telescope Control System keeps a log of all commands issued to the server program. A new file is opened every time the Tcs software is run, thus avoiding the problem of overwriting the previous history file. The TCS files are stored on the Tcs computer in the directory /data/mesg. Because the TCS was restarted numerous times multiple files were created. The files consulted for this report are:
mesg_s92y2005m01d14t0212.tcsmon
mesg_s92y2005m01d14t1511.tcsmon
mesg_s92y2005m01d14t2255.tcsmon
mesg_s92y2005m01d14t2304.tcsmon
mesg_s92y2005m01d14t2312.tcsmon
mesg_s92y2005m01d15t0003.tcsmon
mesg_s92y2005m01d15t0035.tcsmon
mesg_s92y2005m01d15t0036.tcsmon
mesg_s92y2005m01d15t0041.tcsmon
mesg_s92y2005m01d15t0053.tcsmon
mesg_s92y2005m01d15t0114.tcsmon
The Tracker subsystem (Tss) also maintains a log of error and information messages. These are the same messages as the ones sent to the Tss terminal display. Just as the Tcs does, the Tss opens a new file every time it is restarted. The logs are kept on Crockett in the directory /user/het/LOGS/ERROR. The files consulted for this report are:
TS_ERROR_20050114021119.log
TS_ERROR_20050114151111.log
TS_ERROR_20050114225411.log
TS_ERROR_20050114230355.log
TS_ERROR_20050114230940.log
TS_ERROR_20050115000136.log
TS_ERROR_20050115003554.log
TS_ERROR_20050115004108.log
TS_ERROR_20050115005116.log
TS_ERROR_20050115011325.log
It was noted that the Tss logs were lacking in information. In particular, there was no record of which commands were sent either from the remote connection or via the local interface. Only some commands are noted and this appears to occur only if a log call had been added for debugging. The log messages can flash by so quickly on the user interface that it is sometimes difficult for the operator to see them. In addition, it would have been useful to know where the software thought the tracker was located. This would have provided a record of tracker motion during the incident. Such logs would also provide data for engineering analysis of the tracker motor behavior. It is a recommendation of this report that such additional logging be added to the Tss program as long as it does not interfere with the response time of the program.
Personnel in the building at the time of the incident include Chevo Terrazas (Operations Engineer) and Sergey Rostopchin (Telescope Operator). Vicki Riley (Telescope Operator) was called in after 0000 UT (1800 CST) by Sergey and George Damm (Electrical Engineer) was called in at ~0200 UT (2000 CST). All were questioned about their recollections of the events that occurred before, during and after the incident.
The source code for the Tcs and the Tss as well as the Pmac was consulted in order to answer questions that arose during the investigation. A number of tests with the software were run to verify the behavior of the software under various anomalous conditions.
The following sequence of events has been reconstructed from the Tcs and Tss system logs and from the statements of the persons involved. Not all events listed in the logs are given here; many of these events are not connected in any way to the incident in question. All times are given in UT. Civil times are noted in parenthesis.
14 Jan 2005 UT
13:17:23 UT (07:17:23 CST)
Tracker slewed to (0, 0)
13:18:24 UT (07:18:24 CST)
Structure commanded south
13:54:42 UT (07:54:42 CST
Tcs/Tss shutdown
These steps are the normal shutdown steps for the tracker and telescope at the end of the night. The tracker is commanded to the center of the field and the telescope structure is pointed south.
15:12:33 UT (09:12:33 CST)
Tracker slewed to (0, 1000)
15:27:04 UT (09:27:04 CST)
Tracker slewed to (1000, 0)
17:15:02 UT (11:15:02 CST)
Tcs/Tss shutdown
These actions were taken by the day staff to move the tracker. Note that Tcs and Tss were shutdown properly but the tracker was left at X = 1000, Y = 0. This will become important later in the scenario.
22:54:11 UT (16:54:11 CST)
Tcs/Tss started normally for Ops Engineering
22:57:47 UT (16:57:47 CST)
Initialize Search begun
Note that the track motors were not engaged prior to the initialize search request.
23:01:24 UT (17:01:24 CST)
Go Next requested
Tcs reports failure as no next object is loaded yet. Note that the initialize search is still in progress. The hexapods are run first and the system requires 2.5 minutes to complete the initialization of the six hexapod motors.
23:01:32 UT (17:01:32 CST)
Abort Trajectory requested. The initialize search is aborted.
23:01:35 UT (17:01:35 CST)
Tss reports failure to initialize the X axis.
23:01:37 UT (17:01:37 CST)
Go Next requested again.
Tcs again reports failure due to no next object loaded
23:02:08 UT (17:02:08 CST)
Initialize search requested.
Tss reports failure because an emergency condition exists.
23:02:55 UT (17:02:55 CST)
Initialize search requested. No message from Tss.
23:03:07 UT (17:03:07 CST)
Abort Trajectory
23:03:30 UT (17:03:30 CST)
Tcs/Tss closed and restarted.
23:04:55 UT (17:04:55 CST)
Go Next requested.
Tcs again reports failure due to no next object loaded
23:05:02 UT (17:05:02 CST)
Tss reports set position in progress and hexapods complete.
The Go Next command initiated this move. The next trajectory point in Tss is set to (0, 0) when the software first comes up. This point had not been changed when the Go Next command was sent. The tracker however, was at (1000, 0) and thus when the tracker was turned on by the Go Next command is initiated a move to (0, 0).
23:05:12 UT (17:05:12 CST)
Initialize search. The motion from Go Next is still in progress. Tss reports failure to initialize on X, Y, and Rho.
23:05:57 UT (17:05:57 CST)
Abort trajectory
23:06:06 UT (17:06:06 CST)
Tracker slewed to (0, 0)
Note that Tss reports no set position complete state, however, if the move was successful, the tracker would be at (0, 0).
23:06:35 UT (17:06:35 CST)
Go Next requested. Tcs again reports failure due to no next object.
23:06:51 UT (17:06:51 CST)
Initialize search requested
Tss reports failure to initialize all axes.
23:07:41 UT (17:07:41 CST)
Initialize normal
Note that no shutdown command has been sent to Tss since the slew at 23:06:06. The position file still contains the position (1000, 0), thus the encoders are set to (1000, 0) even though the tracker is at (0, 0).
23:07:42 UT (17:07:42 CST)
Go next. Tcs fails with no next object.
23:08:34 UT (17:08:34 CST)
Abort Trajectory
23:10:00 UT (17:10:00 CST)
Tcs/Tss shutdown and tracker restarted normally.
Note that the positions file still contains (1000, 0) even though the tracker position is 0, 0).
23:13:22 UT (17:13:22 CST)
Initialize normal.
Encoders remain at (1000, 0) with the tracker actually at (0, 0).
23:13:51 UT (17:13:51 CST)
Tracker slewed to (0, 0).
Note, the encoders now read (0, 0) but the tracker is actually located at a position of (-1000, 0).
23:15:05 UT (17:15:05 CST)
Go next. Again this fails for lack of a next object.
23:15:25 UT (17:15:25 CST)
Initialize search. Again this fails for lack of track motors control.
23:16:50 UT (17:16:50 CST)
Initialize normal
The encoders are reset back to (1000, 0) while the tracker is at (-1000, 0). The encoders and the tracker are a full 2000 millimeters out of agreement.
23:17:26 UT (17:17:26 CST)
Abort trajectory
23:17:54 UT (17:17:54 CST)
Tracker slewed to (0, 0).
Note that because the encoders are not correct the tracker was sent to (-2000, 0). The tracker almost certainly hit the negative hard stop. The limit switches would have activated and caused a graceful shutdown of the motion but the tracker would have coasted into the hard stops.
23:19:00 UT (17:19:00 CST)
Initialize search
23:19:12 UT (17:19:12 CST)
Go next
Tcs again fails for lack of a next object. Note also that the initialize search would not have completed yet.
23:19:45 UT (17:19:45 CST)
Track to (0, 0)
Again the initialize search is no where near completion.
23:29:52 UT (17:29:52 CST)
Tss reports initialize search complete.
If the initialize was successful, both the encoders and the tracker would be approximate at (-1980, 150). However, if in fact the initialization did not succeed, then the encoders could still have been at (1000, 0). This would have happened because the tracker was left of the negative limit switches and the switch would have triggered during the initialization, stopping the motion of the tracker.
23:19:56 UT (17:19:56 CST)
Tss report the set position has started.
If the previous initialization failed, then we could have tracked into the negative hard stops, assuming the limit switch was not active.
23:30:06 UT (17:30:06 CST)
Abort trajectory
23:30:15 UT (17:30:15 CST)
Tracker slewed to (0, 0).
If the previous initialization failed, the tracker would have again been driven into the negative hard stops. If the initialization was successful, the tracker should be sent home.
15 Jan 2005 UT
00:01:00 UT (18:01:00 CST)
Tcs/Tss is shutdown and restarted.
Sergey Rostopchin (Telescope Operator) arrives on site.
00:15:25 UT (18:15:25 CST)
Structure commanded to CCAS.
For the first time the operators can have an independent view of the tracker location.
00:22:47 UT (18:22:47 CST)
Tracker slew to (1650, 1550) in preparation for stacking.
Sergey reported that the tracker was actually on the left side of the array rather than the right side as commanded. If the encoders were still incorrect, then the tracker would have been at (-350, 1550). This indicates that the initialization at 23:29 most likely failed.
00:24:00 UT (18:24:00 CST)
Sunset occurs.
00:33:40 UT (18:33:40 CST)
Tracker commanded to (1650, 1550) again. However it is already there according to the encoders so no motion took place.
At this point there are no more additional actions that bear noting. The operators continued to test the tracker in a generally safe manner for another 1.5 hours until George was called in at 02:00 UT (20:00 CST). The only comments about this period are that at no time during this ninety minute period was there an attempt to reinitialize the axes. The majority of tracker motion was with the tracker motors however, two slews were attempted successfully and were not a danger to the telescope despite the encoders being incorrect. The Mars video image provided positive feedback of the tracker locations and the slews were short.
The following section details the activities taken by the engineering staff.
Friday Jan 14th 2005 CST
The engineering staff was called for assistance in correcting an x-axis skew. George Damm arrived at 8:00 pm. Sergey Rostopchin (Telescope Operator) and Vicky Riley, whom he had called up earlier to help him, were on site. When George arrived Sergey was attempting to reduce the beam skew by moving the track motors in the Tss interface. Sergey knew that the tracker position was incorrect per the encoder readings. The tracker was up against the negative X limits although the location indicated on the tracker x-y monitor was approximately -200 mm. Sergey was attempting to minimize the beam skew indicated on the monitor by moving the tracker lower x motor using absolute value commands. Repeated attempts were unsuccessful. During this time a lower X track amp fault was generated and cleared. Prior to George’s arrival a lower X negative limit had occurred. When Edmundo Balderrama arrived he and George went up in the JLG and found that the upper X axis was ~30 mm to the positive side of the negative limit switch. The lower X axis was on the negative limit switch. Sergey released the lower X slew brake and they manually moved the slew brake disc to move the lower X carriage to the same location as the upper x (measured from the center pin of the limit switch to moving block on carriage that depresses this switch). When they returned to the control room another move was attempted on the lower X track motor using commands in the Tss window. During these attempts they saw no motion indicated by the lower X encoder but did notice that the lower X track DAC voltage (voltage driving the lower X track driver) was at 7.0 V, the maximum allowed value. The motor current at the lower X track motor driver in the igloo under the primary mirror was measured while motion was being commanded and the motor current was observed to jump to 5.5 A. The max continuous current rating for these motor drivers is 6 A. At this point they shut down for the night.
Saturday Jan 15th 2005 CST
In the morning they believed (incorrectly it turns out) that there was no mechanical issue with the lower X motion system since they were able to easily move the slew motor disc when the brake was disengaged. They verified that the motor controller was indeed forcing 5.5 A through the motor by measuring current at the motor. A spare track motor was taken up to the tracker; they disconnected the lower X track motor and the spare motor was hooked up (electrically only). They verified that the track motor controller was working properly since they were able to command the spare motor to turn. A measurement of the motor winding resistance on the spare motor at 6 ohm compared to the windings on the lower x track motor at 3 ohm lead them to conclude that they had a problem with the lower X track motor and it was removed from the tracker.
The lower X track motor was tested on the ground and found to operate correctly. They also found that the motor impedance of the upper X track motor was ~ 2 ohms. At this point they began to doubt the original diagnosis of a bad lower X track motor. Edmundo and George returned to the lower X track motor mount. The lower X track brake was released and they attempted to rotate the lead screw. In a properly working system, the torque required for rotation would be the same as that required to rotate the slew motor disc when its brake was disengaged. Both Edmundo and George found that they were barely able to move the disc. George estimates at least 50 ft-pounds of torque was required. As a reference, they loosened the upper X track motor to lead screw coupling to measure the torque required for rotation. They found this brake disc difficult to move but very much easier than that of the lower X track system. When working on the lower X tracker brake they noted that the brakes were visually released from the disc when requested so they concluded that the problem must be with the lower X lead screw bearing system. They stopped work at this point with the intention of having Robert Poenisch come in on Sunday to perform a bearing replacement on the lower X lead screw. John Booth was called and informed of the decision. He reviewed the logic that led them to this decision and concurred with their conclusion.
Sunday January 16th 2005 CST
Robert and Edmundo performed an initial inspection of the lower X lead screw system in the morning. Robert found that the track brake’s resistance to turning was due to the edge of the steel brake disc rubbing against a chamfer on the aluminum housing holding the brake caliper. It was originally assumed that this misalignment was due the inability of the caliper to float axially along the lead screw to maintain optimum position relative to the brake disc. Robert cleaned the pins which the calipers float on and was able to free this stuck mechanism. He realigned the assembly on the lower X track motor mount and then he and Edmundo reinstalled the original lower X tracker motor.
Edmundo and Robert next aligned the upper and lower X axes by releasing the slew motor brake and rotating the disc. The tracker was aligned using a ruler to measure the position relative to the fixed limit switch on both the upper and lower X travel.
At this point an attempt was made to move the tracker using commands in the Tss window. It was noted that as soon as the 3-phase power was turned on the position update window in both Tss and the X-Y state monitor no longer updated positions. After several attempts, a decision was made to wait for Jim Fowler’s input before proceeding. Contact was made with Jim that evening. He talked with Bob Calder and suggested a couple of tests. Bob attempted these tests with negative results.
Monday January 17th 2005 CST
Jim arrived early Monday afternoon and after a review of the logs he powered up the tracker and began the usual initialization sequence. The tracker worked normally during this initialization. Upon further discussions between Jim and Bob it was determined that the problem was a procedural error in the Tracker Slew Recovery procedure. In essence, when the Tss was brought up the Pmac card did not yet have closed loop control of the motors. Thus when the low level motion commands were tried the Pmac did not command any motion and the encoder positions did not change. This procedure should be changed to allow the slew recovery procedure to operate correctly. The tracker was tested, deemed operational and was available for science on the night of January 17th CST.
Tuesday January 18th 2005 CST
A verbal review was held with all staff to discuss the events and procedures used over the weekend. The review and additional investigation resulted in this report.
Paul and Robert took an additional look at the lower X track motor brake assembly in order to perform the required alignment. Close inspection revealed that the problem was not with the motor brake but with the alignment of the bracket that holds the track motor, brake and lead screw bearing. All three elements are mounted on a common bracket as shown in Figure 1.
Figure 1. Diagram showing mounting of track motor, brake and bearings for the X axis lead screw system. Both the upper x and lower x systems are identical. The X axis carriage is mounted to a nut that moves along the lead screw.
Paul found that the both the motor bracket and the bearing bracket on the opposite end of the lower x lead screw showed a clockwise rotation when viewed from the bottom end of the tracker. This displacement is consistent with the tracker being driven into the negative hard limit. Once the negative hard limit is engaged and the system continues to attempt further negative motion, the bracket at both ends of the lead screw will be pushed to the right. This motion also resulted in a misalignment between the disc brake and the caliper leading to binding. Verification of the ‘hard limit crash’ event was also seen by displacement of both brackets on the upper X lead screw.
During the past summer (2004) the upper X track motor had been adjusted to reduce a misalignment seen at the motor coupler. During this process, marks were made on both sides of the motor bracket. From these marks it was clear to see that the motor bracket had been displaced as described.
The corrective action taken by Paul and Robert was to loosen the bolts holding the brackets at both ends of the lead screw. Shims previously placed under the motor mount were removed. At this point the brackets were realigned to place the lead screw in line with the track motor. The bracket bolts were then retightened.
The Operations Engineer at HET is tasked with the job of insuring that all operation’s systems, i.e. structure, tracker, dome, etc, are in proper working order prior to the observing staff arrival at sunset. This activity checks to see that the engineering staff has not left equipment in a non-operational or non-normal operational mode as the result of engineering work during the day. If problems are found, the Operations Engineer should attempt to fix simple problems and should contact the responsible engineer in the event that a serious problem is found. The Ops Engineering Procedure and Troubleshooting Guide is located at http://het.as.utexas.edu/HET/OpsMan/toc.htm (see the 2004 documents by Martin Villareal.) The document of interest for this investigation is the Tcs/Tracker Initialization page (also available in Word format.) This page describes the steps required to properly initialize the Tcs and Tss software and hardware. Although there are some minor errors on this page, they are not related to this problem. The initialization procedure is correct and will work properly as currently written.
The basic steps of the initialization process are, first, visually identify the location of the tracker; second, start the software and power up the tracker electronics; third, enable the track motors, and fourth, run the initialization sequence. Step 1.C of the procedure specifically calls for identifying the location of the tracker. This step can also be done during the interior inspection step of the overall Ops Eng procedure.
Steps 2.1 and 2.2 describe how to power up the track motors. This step is crucial because the PMAC initialize routine will not move the X, Y, or Rho motor unless the Pmac has taken closed loop operation of the motors at least once prior to the start of the routine.
This problem does not occur on the hexapod axes because the hexapod motors are brought under closed loop control at the time the tracker is powered up. However the X, Y, and Rho motors are in the initially powered down state at startup. This known feature of the Pmac cards and the HET control software has been a part of the system since operations were started.
The best procedure to take control of the motors is to use the Tracker Position… menu option under the Tools menu of Tcs. Selecting this will bring up a position dialog with the X and Y values set to the current position of the tracker. The Z value defaults to 60.5mm, the normal shutdown location, while Theta, Phi and Rho default to 0.0 degrees. When one selects the Ok button on this dialog, a second dialog is brought up asking if you want to use the Tracker or Slew motors for the move. In this case the track motors should be selected. Simply selecting Tracker position, clicking on the Ok button and selecting the track motors should be enough to send a command to enable the motors with out moving the tracker. This is enough to allow the initialization procedure to complete successfully.
Steps 2.3 through 2.9 detail the initialization procedure. Step 2.4 refers to a dialog box that is opened when the Initialize Tracker… menu option is selected. This dialog box contains two options, Initialize Normal and Initialize Search, along with six grayed boxes. These boxes indicate the status of the initialization state of the three hexapod pairs along with the X, Y, and Rho axes. A grey box means no initialization has been attempted since the start of the software, a yellow box means an initialization of the axis is in progress, a red box means that the last initialization attempt failed, and a green box indicates that the last initialization attempt was successful. All six boxes must be green before the initialization is considered successful. The entire initialization process requires about 8 minutes to complete.
The Initialize Normal button on the Initialization Dialog does not move any motors but does set the encoder position counters to the last position saved in the file crockett:/user/het/POSITIONS.INI. This file is written when ever a tracker shutdown command is executed and must be done before the Tss software can properly exit. The current encoder position of the six Hexapods, Rho, Y, lower X and upper X as well as the current trajectory point are saved in this file.
In addition to the Initialize Dialog, a separate Dialog box comes up when the Tcs software is started. This box also allows one to run the Initialize Normal or Initialize Search options and in addition allows one to Take Control of the tracker. The Take Control option simply sets the time in the Tss software but does not move motors or change encoder readings. This is the default option for the Startup Dialog.
A review of the system logs, as described in the section Sequence of Events, show that at no time during the event was the Tracker Position option used to power on the motors. Instead the Operations Engineer used the Go Next button on Tcs to power on the motors. The Go Next button is normally used to send the tracker to the start of a new trajectory as described by the internal Tcs variable next_obj. However, if the next_obj structure is empty, then no new position is sent to the tracker. But the motors are enabled, powered up and commanded to the current trajectory point. Usually the current trajectory point is the current position of the tracker so this is not a problem and it requires fewer steps that the Tracker Position option. Therefore, this has become a short cut for the Telescope Operators and some Operations Engineers to use in lieu of the Tracker Position option. There is a danger in this however, in that if the tracker is not at the position given in the current trajectory point of the Tss, then the Go Next button will cause the motor to move the tracker in an attempt to reach the current trajectory point. This action is the most likely cause of the tracker motion at 23:05:02 UT.
Because the Go Next operation can cause unintended motion it is not recommended for powering up the motors. Safety problem can arise if the tracker moves when not expected. For this reason the Go Next button should do nothing if no valid next object exists. It is the recommendation of this report that the behavior of the Go Next option be changed to reflect this.
The tracker x-axis permits a physical range of motion of a little over 4000 mm, ±2000 mm about a centrally located zero point. The tracker x motion is driven by 2 lead screws, upper-x and lower-x. Both are synchronized together by Pmac motion control cards, using Heidenhain encoders at both upper and lower x to maintain coordinated motion and to minimize skew. Skew is the difference between the position of upper and lower x positions. A software skew limit of 75 mm, once breached prevents x axis motion. A hardware skew limit of 100 mm prevents motion in the event that the software limit breach does not stop x axis motion. In the worst possible scenario, an uncontrolled or large skew will cause the tracker to fall from the top of the structure.
The x motions on both upper and lower lead screws are driven by two separate motors. The slew motor drives the tracker at 80 mm/s. This 3 phase brushless DC motor has a lead screw nut locked to the rotor of this frameless motor. The lead screw runs axially down the center of this motor. When the rotor spins and the lead screw is held stationary, the nut and the attached tracker translate along the x axis. The track motor drives the tracker at speeds up to 3 mm/s. The track motor is a brushed DC motor that is physically connected to the lead screw shaft. When the previously mentioned roller screw nut in the rotor of the slew motor is held stationary (using a disc brake attached to same) and the lead screw is rotated by the track motor, the tracker will translate along the x axis. The slew motion mode is used to move the tracker quickly within its operation space. The track motor is use to track astronomical targets.
The tracker is protected from traveling too far by both software and hardware limits. In addition software and hardware monitor the skew, i.e. the difference in position, between the upper and lower X axes.
The first monitor on the travel limits is supposed to be a software limit programmed into the Pmac memory. The `I’ variables Ix13 and Ix14 (where x is the motor axis number) are the positive and negative limits respectively. These two limits are used by the Pmac to stop the tracker motion if a motor moves beyond these limits. For the tracker motors these limits are set to ±1951 mm while for the slew motors these limits are set to ±1646 mm. When the Pmac detects that a motor has passed one of these limits it initiates a controlled deceleration of all the running motors. The deceleration parameter is defined in the variable Ix15. For the track motors this values is set to 0.5 mm/s2, which for a maximum track speed of 3 mm/s results in a deceleration time of 6 seconds and a deceleration distance of 9 mm at maximum speed. For the slew motors this value is set to 20 mm/s2, which for the maximum slew speed of 80 mm/s results in a deceleration time of 4 seconds and a deceleration travel distance of 160 mm at maximum speed. However, these positions are based on the encoder readings. If the encoders are reset and contain incorrect values, then the Pmac will not be aware that it has moved a motor past the limits in the absolute sense. The Pmac will only sense this error when the encoder reaches this distance with respect to the current encoder zero point.
Since the software limits are only relative measurements, physical micro switches have been installed on the tracker assembly at either end of the tracker travel range. These switches are connected to the hardware limit inputs of the motor axis on the Pmac card. The switches are normally closed so that when the Pmac detects that one of the switches has opened it causes all motors to decelerate in a controlled fashion. The deceleration is controlled by Ix15 just as in the software limit. Thus the track motor deceleration requires 6 seconds and covers 9 mm, while the slew motor deceleration requires 4 seconds and covers 160 mm. The switches are located at ±1880 mm (± 3mm variation).
Note that these limit switches are located inside the previously mentioned software limits.
The software limits would not have triggered before the limit switches. It is a recommendation of this report that the software limits be changed to ±1870 mm or that the hardware limit switches be moved.
Finally if the tracker travels over the limits switches and continues to travel for whatever reason, a set of rubber bumpers, called hard stops, exist and the tracker will strike one or more of these. Striking a hard stop is very hard on the tracker as it causes the approximately six tons of weight to come to an abrupt halt. The upper and lower negative hard stops are located 87 mm and 93 mm respectively past the limit switches.
Note that this is less than the 160 mm run out of the slew motors. If the tracker is running at full slew speed when it reaches a limit switch (and this is almost certainly the case), then if would not have had time to decelerate before striking the hard stops.
Note that the limit switches that control the track motion limits also control the slew motion limits. This means that if the encoders are incorrect and slew motion occurs beyond the software limit of 1645 mm, then the tracker does not have enough time to halt the slew motion in the event the limit switches are triggered and the tracker may strike the hard stops. Had limit switches also been installed at ±1700 that were connected to the slew motor limits on the Pmac, then no negative slew motion could have been commanded when the tracker was physically left of these limits. Such a switch mechanism could be implemented by using a toggle style switch that would not require a long drag plate to keep the switch active over 300 mm of travel. It is a recommendation of this report that a system for such switches be designed and installed on the tracker.
It was noted during this investigation that the X and Y encoder tapes provide an absolute distance encoding which is used by the tracker software to determine the location of the tracker during initialization. This encoding is provided by a set of index marks which change their relative spacing in a known manner. By recording the two distances between three index marks it is possible to monitor the current tracker position independently of the quadrature encoder marks used to monitor the position. It may be possible to implement software on the Pmac boards or within the Tss program that continually monitors for index marks and calculates a position that could be compared to the current encoders. Such a system should be explored and the cost/benefits considered.
The direct cause of this incident was the loss of encoder positions by the use of Initialize Normal at an inappropriate time, a slew command while too close to the limit switches, and the lack of properly configured limits in the tracker. Secondarily this was caused by a lack of experience on the part of the Ops Eng. Chevo reports that this was only the third time he had done Ops Eng and it was the first time he tried to do so with out following the step by step procedure. He has only worked at the observatory for less then four months. Indeed the entire night staff that evening consisted of the newest and least experienced members of the staff. We do not mean to imply that lack of attention was a problem. Only that experience counts for quite a lot when problems arise and inadequate safeties are in place.
The following actions, listed in order of priority, are recommended to prevent a recurrence of this incident and to provide better data collection in the event of future incidents.
Change the software limits Ix113/Ix114 in the Pmac to be ±1870 mm or move the limit switches.
Add additional logging to the Tss program. All commands should be logged as well as any error conditions, e.g. limit switch activation.
Install additional limit switches at ±1700 on both X axes and the Y axis to provide the correct limit indication to the slew motor.
Add engineering logging to the Tss program. Review data to include in logs, e.g. commanded positions, actual positions. Such logs will provide improved information about the software’s knowledge of the tracker position and will provide information required to track motor performance.
Modify Tcs Go Next button to do nothing if no valid object exists.
Revise Skew Recovery Procedure to account for the situation were the motors are not yet ready to be commanded.
Provide additional training to TOs and engineering staff about tracker operations.
Review requirements for implementing a procedure to do real time correction of the position recorded by the encoders by monitoring the position of the index marks. Determine the amount of work required to implement such a system and if such a system would provide improved performance and safety.
Devise procedure and installation requirements for installing hard limit switches between the current limit switches and the hard stop.