On the night of 15-16 May 2000, during an engineering run, the HET tracker experienced a situation in which the Y-slew motor was off and the slew brake was open. This allowed the tracker carriage to back-drive along the Y–lead screw for approximately 3 meters until it hit the hard stop bumpers at the base of the tracker. The carriage bounced once off the hard stop and then came to rest on the bumpers.

A physical inspection of the tracker and carriage by John Booth, Craig Nance, Tom Worthington, and Ben Laws, carried out just after the incident, showed no physical damage to any mechanical or electrical assemblies. Neither did they find any indication of why this event might have occurred. Bill Spiesman checked the state of the low-level software approximately one hour after the incident occurred but could not find anything that might have caused the problem. Jim Fowler reviewed all available evidence as well as the system documentation and prepared this report.

It was not possible to determine the exact cause of the incident based on the available evidence. However, a review of the software and hardware design showed at least four situations where the system would believe that the amplifier was able to move the motor when in fact it would not. The following scenario is considered the most likely sequence of events that led to the incident.

During the day, the staff lubricated the X and Y lead screws. The tracker was slewed successfully during this work. When the lubrication was completed, the tracker motors were turned off but the software and control power were left on. During the 5.5 hours that elapsed before the system was used again a number of brief power fluctuations occurred. These fluctuations caused the DC power to the amplifiers to fail or to drop below the value required to power the motors. Because such a situation is neither a power supply nor an amplifier fault, no indication of the problem was reported to the system. When the Telescope Operator arrived, the software and hardware were not restarted, either by the TO or by the Operations Engineer, which would have cleared the condition. The first command to the system was an ‘initialize’, which only turns on the track motors. When the slew command was issued, it resulted in the Y-slew amplifier not turning on, but not indicating a problem to the PMAC. The PMAC then released the slew brake and the carriage traveled down the lead screw until it hit the hard stop.

Back driving along the Y-lead screw until the carriage hits the hard stop is considered the second worst problem that could occur on the tracker. A number of mechanisms were thought to be in place at numerous levels of the software and hardware to prevent such an incident. These mechanisms were either not in place or did not perform as expected during the incident. In short, the Y-slew system as delivered and operated was not fail-safe.

The following actions, listed in order of priority, are recommended to prevent a recurrence of this incident and to provide for better data collection in the event of future incidents.

• Install warning and fatal following error limits in all three slew motor controllers.



• Add a DC voltage monitor at the slew amplifiers that is connected in parallel with the amplifier fault line to the PMAC or consider the use of the Drive-Up contacts in place of the amplifier fault contacts.



• Modify operations procedures to shut down the tracker and telescope control software if not being used for any length of time. Until the software is more robust this should be a routine practice.



• Modify the Tcs GUI to provide better bounds checking on Set Position data values. Insure that this check is always made.



• Modify PLC 20 to provide a test for amplifier activity between activating motor control and releasing the brake. Change the order of brake release and set.



• Modify the tss software on Crockett to provide a new log file at every startup.



• Create a PMAC dump command that records all information regarding the current state of a PMAC controller.



• Install an incoming power monitor to report power fluctuations.

On the night of 15-16 May 2000, during an engineering run, the HET tracker appeared to experience a situation in which the Y-slew motor was off and the slew brake was open. This allowed the tracker carriage to back-drive along the Y–lead screw for approximately 3 meters until it hit the hard stop bumpers at the base of the tracker. The carriage bounced once off the hard stop and then came to rest on the bumpers.

A physical inspection of the tracker and carriage by John Booth, Craig Nance, Tom Worthington, and Ben Laws, carried out just after the incident, showed no physical damage to any mechanical or electrical assemblies. Neither did they find any indication of why this event might have occurred. Bill Spiesman checked the state of the low-level software approximately one hour after the incident occurred but could not find anything that might have caused the problem. Jim Fowler reviewed all available evidence as well as the system documentation and prepared this report.

The format of this report follows the flow of information from the Telescope Operator’s request, through the software, ending up in the hardware. As was pointed out by the initial inspection the night of the incident, there were no indications of what caused the problem. Thus it was necessary to examine the entire system, determine what problem areas might exist, and find out what might have caused the problem given the conditions at the time of the incident. Section one provides a description of the information sources used to determine the equipment status at the time of the incident as well as to find out how the system behaves. Section two describes the sequence of events that occurred as reconstructed from the logs and personal statements of the staff. Section three is a description of and a commentary on the software used to command a tracker motion. Section four describes the hardware system and how the various signals control the motors. Based on this information Section five provides a best guess as to the cause of the problem. A list of recommended changes and additions is provided in Section six. Finally, some specific questions that were raised during the initial incident are answered.

Data logs and information about the state and history of the Tracker system as well as the Telescope Control System (Tcs) are available in a number of locations. The Telescope Control System keeps a log of all commands issued to the monitor program. A new file is opened every time the Tcs software is run, thus avoiding the problem of overwriting the previous history file. The Tcs file is stored on the Tcs computer in the directory /data/mesg. The file in use at the time was mesg_s92y2000m05d15t1932.tcsmon. This file was opened at 1932 15 May 2000 UT (1432 15 May 2000 CDT) and closed at 0606 16 May 2000 UT (0106 16 May 2000 CDT).

The Tracker program (tss) maintains a log of error and info messages. These are the same messages that are sent to the tss terminal display. Unfortunately this file is overwritten every time the tss program is restarted. The file in use at the time of the incident was overwritten later in the night, approximately 3.5 hours following the incident. The tss program should be modified to write a new file every time it starts.

Following the incident Bill Spiesman, who was working at the Harlan J. Smith telescope at the time, was contacted. He reviewed the state of the PMAC and recorded the I- and M- variables associated with the Y-track and -slew motors. These data were collected at ~0340 16 May UT (~2240 CDT 15 May). Unfortunately, these data do not indicate any problems with the PMAC system. Because the PMAC and amplifiers had been reset during the additional attempts to move the tracker most of the indicative data had been cleared. It would be useful to create a program that dumps and saves the PMAC state information so that the Telescope Operators can quickly and easily record the state of the PMAC without requiring a detailed knowledge of the PMAC system.

Personnel in the building at the time of the incident include Gabrelle Saurage (Telescope Operator), Matt Shetrone (Resident Astronomer), Craig Nance (Facility Manager), John Booth (Mechanical Engineer, Austin), Edmundo Balderrama (Electromechanical Technician), Ben Laws (System Analyst) and Brian Roman (Telescope Operator). Tom Worthington (Mechanical Engineer, HET) was called in from home to assist in the inspection process. All were questioned about their recollections of what occurred before, during, and immediately after the incident.

The source code for the Tcs and tss software was reviewed. No specific problems were noted though a number of improvements can be made. At least one change to the PMAC software, enabling fatal following errors, while not capable of preventing this incident, would have stopped the tracker from back-driving all the way down the lead screw.

All electrical drawings relevant to the Y-slew circuitry were reviewed and the wiring checked for compliance. The operating manual for the Y-slew amplifier was reviewed. A number of tests were performed with the hardware to verify expected operations and to test various problem scenarios.

This is the sequence of events as reconstructed from the statements by the staff and the program logs. All times are given as UT. Central Daylight Time noted in parenthesis. This chronology has been reconstructed primarily from the Tcs log and personal statements.

Note the long period of time between the last motion and the next command. Almost 5.5 hours elapse between commands. It has been my experience with the Tcs/Tracker interface that the links between them invariably crash when they have been inactive too long, on the order of 1-2 hours is typical. It is very likely that these links had died during this period. How this may have affected the tracker operation is unclear at this time.

Edmundo also reports that between 2200 – 2400 UT (1700 – 1900 CDT) there were several brief drops on the incoming power lines. These were reported as “power blinks”.

The Tcs monitor command that was used to move the tracker is Set Position. This command is typically used for single motions of the tracker. What follows is a description of how the Tcs and tss software manipulate and transmit the position data to the hardware.

During a Set Position request, the position data is entered into the Set Position dialog box on the Tcs computer. The Set Position dialog consists of six text boxes, one for each of the tracker axes. If the user enters data in one box and then selects another text box on the dialog, the software is suppose to compare the data entered to make sure that no invalid data has been entered as well as to check if the numerical values are within limits. This feature appears to work the first few times the dialog box is opened, however, it appears to fail after the dialog has been used 3-4 times. Once the checks are no longer made it is possible to enter complete garbage in the field, which may parse to a valid but unknown position. Randy Ricklefs is looking into this behavior. In addition the limits that are used to verify the position are compiled into the software and are set very wide. The x/y limits are ± 5000 millimeters, the z limits are ± 500 millimeters, the rho limits are ± 360 degrees and the theta/phi limits are ± 20 degrees. These limits can be found in tcsgui/CreateForms.c:CreateTkrPositionForm(). Such wide limits provide no check at all and should be narrowed to more appropriate values.

Once the data is considered ok it is compared to the current position of the tracker and the user is told how far the system needs to move and how much range is left on the slew systems. The user then selects the speed for the move. The position and speed information is sent to the Tcs monitor using the tcsmon/command.c:tkrposition() command. This command reads the command string and compares the values to the limits defined in /data/lib/tracker.dat. Relevant values are KSDx, KSDy, KSFz, KSFtheta, KSFphi, and KSFrho. The current limits on y in this file are ± 1900 millimeters. If all values are within limits, then the position is passed to tcsmon/rdscope.c:move_tkr(). This function adds all mount models, loads the positions into shared memory, and triggers the tracker communications process to send the position to the tracker.

The position data is received by the net_exec process on the tracker and passed to net/netproc.c:process_set_postion(). For a slew request, the position is compared to the limits set on the X/Y/Rho PMAC board for the slew motors. For the Y motor these are located in variables I613 and I614, which are set to ± 1646.2 millimeters. The function then commands the X/Y/Rho PMAC to coordinate system 2, (the slew coordinate system), sets the tracker display to show ‘S’ to indicate the slew motors are in operation, and runs the PMAC motor initialization program (PLC 20).

PLC 20 first kills all current motor activity, releasing the motors from closed loop control, and sets variable P10 to indicate to the other PMAC programs that the motors are not ready to run. Any previously latched amplifier faults are cleared. The maximum permitted velocity (Ix16, 80mm/s), acceleration (Ix17, 20 mm/s²), and limit-switch deceleration (Ix15, 20 mm/s²) values are set. All motors are enabled, the slew coordinate system is selected and closed loop control is taken on the x/y slew motors as well as the rho motor. This step should enable the amplifier and apply power to the motor windings. The slew brakes are released and the track brakes are set. Note that there is no pause between commanding the motors on and the release of the brakes in order to allow the amplifiers to power up prior to releasing the brakes. Nor is there any test to verify that the amplifiers have correctly powered on. In addition, the sequence of releasing the slew brakes prior to setting the track brakes is backwards, as all brakes should be set before any are released. This is a result of copying a section of code from a previous portion of the program. However, I don’t feel that this had anything to do with this incident. Finally the rho brakes are released, the program pauses for one second, then clears the rho pull-brake.

The tss program then applies the native mount model corrections, the hexapod leg lengths are calculated, the updated positions are placed in the dual-ported RAM memory, and the PMAC motion program, program 8, is run to move the tracker. Motion program 8 sets the acceleration time, s-curve time, and motion times, then starts the motion with an axis move command based on the position in Mx51. The acceleration, s-curve, and motion time are currently set to 1000 milliseconds, 500 milliseconds, and 1000 milliseconds respectively. Note that these values may be too fast for the tracker system. These are the default values used by Delta Tau for their small motors and are probably not applicable to our larger system. However, they are over-ridden by other variables in the PMAC system and are probably reset here to prevent residual large values from controlling the motion.

Although a number of areas in the software systems need updating and correcting, there does not appear to be any portion of the software that would explicitly allow the slew motor to be commanded to an invalid position at an invalid speed.

The tracker contains two Programmable Multi-Axis Control (PMAC) boards, manufactured by Delta Tau Corporation. One controls the X/Y/Rho motors (both track and slew), and one board controls the six hexapod motors. This report describes only the X/Y/Rho board.

The PMAC boards are plugged into the tracker computer system bus and receive new position information through the dual ported ram. The positions are stored in variables Mx51. The motion program that is used to do a Set Position request (program number 8) uses the Axis Motion command, a built-in function of the PMAC control system, in order to move the motor. Essentially the data in Mx51 are used to define the endpoints of a move. Various variables within the PMAC control system control the velocity, acceleration, and total move time.

The PMAC board provides an enable signal to the motor amplifier that tells the amplifier to turn on power to the motors. It also provides a velocity control signal that tells the amplifier how fast to run the motor. The amplifier provides a fault indication to the PMAC board. This fault indication is an open-collector output which grounds the signal line if an amplifier fault occurs. For the HET Y-slew system, an amplifier fault occurs if the amplifier detects an over-current or over-voltage condition. A test of the Y-slew amplifier showed that the amplifier is capable of generating an amplifier fault and that the PMAC responds correctly to this signal. Note that if there is no 120 VAC control voltage applied to the amplifier then there can be NO amplifier fault signal.

During the motor initialization (PLC 20) the PMAC board sets the enable line low unless an amp fault is detected. When the amplifier receives the enable signal it turns on power to the motor coils. Once the PMAC has set the enable line it releases the associated brake. When the Axis Motion is requested, the velocity control signal is sent to the amplifier and the motor is moved. The PMAC tracks the position of the carriage. As the motor gets close to the requested position the PMAC slows the velocity and stops the motor gently. The safety program, PLC 30, monitors the open-loop bit of the motors. If it detects that a motor is open-loop, that is, the PMAC is no longer commanding the amplifier and the enable line is off, then the associated brake is set for the motor.

Besides the amplifier fault line, the only other indication the PMAC has that there is a problem with the motor system is the following error indication. The following error is defined as the difference between the commanded position of the motor and the actual position. If the following error exceeds some predetermined limit, the control system will stop any motion in progress under the assumption that something has gone wrong in the control system.

Within the X/Y/Rho system the fatal following error limit is set in motor variables Ix11 and a warning following error limit is set in Ix12.During tracking, the track motor fatal following error limit is set to Ix11 = 1219200 or 38.1 mm (I-variable units are 1/16 of an encoder count) while the warning following error limit is set to Ix12 = 38100 or 1.2 mm. However, all the slew motors have both the warning and the fatal limits disabled.

Delta Tau, the manufacturer of the PMAC boards, specifically recommends against disabling the following error limits.The documentation in the PMAC Software Manual for Ix11 states, “Setting Ix11 to zero disables the fatal following error limit for the motor.This may be desirable during initial development work, but it is strongly discouraged in actual operations. A fatal following error limit is a very important protection against various types of faults, such as loss of feedback, that can not be detected directly, and that can cause severe damage to people and equipment.”

The HET tracker PMAC software includes an initialization program called init_1.plc. This program is called when the main tss program starts. Within this program are explicit statements that set I311 (upper X slew), I411 (lower X slew) and I611 (Y slew) to zero. Todd Bjerke added the following errors to the program on 13 June 1996 (Revision 1.3), according to the revision comments. These same comments also indicate that the slew motors limits were set to zero on 16 June 1996 (Revision 1.6) at the request of the Orbital GNC group (Guidance, Navigation and Computing). Whether this was for initial development or operations is not known.

My suspicion, based on my experience with the tracker system, is that the slew motors’ following error limits were set to zero for initial development because the motor tuning was very poor. This could have created numerous following errors that would hamper testing. These limits have never been re-enabled.

The following errors in all three slew systems have been re-enabled with the same value as the track motors. The fatal error occurs at 38 mm of following error.

A careful review of the logs, the personnel statements, the software, and the hardware manuals provided no “smoking gun” pointing to the exact cause of the problem. With the available data, it is not possible to say exactly what occurred before, during, and after the incident. Only an informed guess as to the cause of the problem is possible.

Although the amplifier provides a fault signal back to the PMAC board, a review of the amplifier operations manual reveals that there are at least three situations that would prevent the amplifier from operating the motor but that would NOT provide an amplifier fault indication to the PMAC. This first is an under-voltage on the DC bus. DC power is provided by an external power supply, which has fault contacts wired into the emergency stop system. However, if the incoming three-phase power is bad and the power supply is not otherwise at fault, these contacts will remain closed. If the DC voltage at the amplifier is low, the amplifier will not turn on when the enable line is activated. However, since no amplifier fault is generated, the PMAC has no knowledge of this problem and will quite happily release the slew brakes. With the motor off and the slew brake released the tracker carriage will back-drive along the lead screw. With no following error limit set the carriage will back-drive until it hits the lower hard stops.

It was possible to test this scenario by removing the three-phase input that powers the DC bus. Control voltage was still available to both the amplifier and the power supply. The amplifier front panel lights did indicate an under-voltage condition but it did not trigger the amplifier fault signal. The power supply front panel also indicated that the DC voltage was not ready but the power supply fault lines were not triggered. Following this the air supply to the Y-slew brake was removed to prevent the carriage from back-driving. Under these circumstances it was possible to command a slew motion without any errors and the PMAC system commanded the slew brake off even though the amplifier was not providing power to the motor.

A loss of AC control voltage to the amplifier would result in the same situation. The amplifier requires 115 VAC to provide power to the logic and control circuits. The amplifier fault circuit provides an open collector signal that provides a low resistance path to ground when an amplifier fault occurs. If there is no control voltage to the amplifier, it is not possible to turn on the amplifier fault signal. Since the amplifier fault signal is the only indication the PMAC has that the amplifier is ready, it cannot detect whether the amplifier is on or not. Indeed, the amplifier could be completely removed and the PMAC would not be aware of the fact.

The slew motor provides a Rate Position Sensor signal back to the amplifier. The amplifier uses this signal to operate its own feedback control loop. The loss of this signal causes the amplifier to be unable to control the motor, and once again no amplifier fault signal is provided to the PMAC.

This particular scenario was also tested on the tracker. With following errors enabled and the tracker at a Y value of –1600, the Rate Position signal was removed from the amplifier. When the amplifier was enabled it was unable to control the motor and the carriage back-drove down the lead screw. Motion was halted only by the activation of a fatal following error after 38 mm. From a standing stop the carriage traveled a total of 100 mm before coming to a halt

If the amplifier detects an over-temperature condition, it will shutdown the amplifier until the temperature reduces and the reset line is pulled to ground. Again no amplifier fault indication occurs.

The operations manual indicates that there is a set of contacts available on the amplifier known as the Drive-Up contacts. Although not entirely clear from the manual it appears that these contacts can be set to open if the amplifier detects any fault, including DC under-voltage, loss of control voltage, and over-temperature. It is not clear if it will also detect a loss of the Rate Position signal. This contact should be considered in place of the amplifier fault contact as the control signal from the amplifier to the PMAC.

Based on the above information the following scenario is considered the most likely sequence of events leading up to the incident. During the day the staff lubricated the X and Y lead screws. The tracker was slewed successfully during this work. When the lubrication was completed the tracker motors were turned off but the software and power were left on. During the 5.5 hours that elapsed before the system was used again a number of brief power fluctuations occurred. These fluctuations caused the DC power to the amplifiers to fail or drop below the level required to power the motors. Because such a situation is neither a power supply nor an amplifier fault, no indication of the problem was reported to the system. When the Telescope Operator arrived, the software and hardware were not restarted, either by the TO or by the Operations Engineer, which would have reset this condition. When the slew command was issued following the initialization, it resulted in the amplifier not turning on, but not indicating a problem to the PMAC. The PMAC then released the slew brake and the carriage traveled down the lead screw until it hit the hard stop.

The following actions, in order of priority, are recommended to prevent a recurrence of this incident and to provide for better data collection in the event of future incidents.

• Install warning and fatal following error limits in all three slew motor controllers. Verify normal following errors and adjust accordingly. This has been completed.
• Add a DC voltage monitor at the Y-slew amp that is connected in parallel with the amplifier fault line to the PMAC. The exact voltage limit will need to be determined.
• Modify operations procedures to shut down the tracker and telescope control software if not being used far any length of time. Until the software is more robust this should be a routine practice.
• Modify the Tcs GUI to provide better bounds checking on Set Position data values. Insure that this check is always made.
• Modify PLC 20 to provide a test for amplifier activity between activating motor control and releasing the brake. Change the order of brake release and set.
• Modify the tss software on Crockett to provide a new log file at every startup. These files will need to be culled periodically from the tracker computer so that the available disk space is not entirely used up.
• Create a PMAC dump command that provides all information regarding the current state of a PMAC controller.
• Install a power monitor on the incoming power to report power fluctuations.

Following this incident a number of questions were raised in the email messages that I saw. I would like to answer those questions here.

1. Why was there no fatal following error? See the section under Hardware Control Sequence for a description of why no fatal following error occurred.
2. An initialize search was done. Why did this not clear the problem? During the initialize search sequence the track motors are used to move the tracker until the position markers are found. The most likely cause of this incident was a loss of DC power to the slew amplifiers. The tracker motors receive their power through a separate system and thus they worked just fine.
3. Was the tracker commanded to (1600, 1550) as the TO reported or was it commanded to (1600, -2000)? The tracker was commanded to (1600, 1550) as is clearly indicated in the Tcs log.
4. Would the tracker accept a command to (1600, -2000) as a valid input? No. Although some of the input tests did not work correctly other tests would have caught this as an invalid input and would have refused to move the tracker.
5. Why didn’t the software limit at Y= -1650 stop the tracker as usual? The tracker would have attempted to decelerate the Y-slew motor when the carriage reached Y = -1650 however, since it had no control over the unpowered motor the carriage continued to travel until it hit the hard stop.
6. Why did the carriage travel much faster than I have heard before? The motor was not powered and the carriage was back-driving under the influence of gravity. It continued to accelerate until limited by the friction of the lead screw and slew motor. Given the time estimates reported by personnel on site at the time, I estimate that the carriage was traveling close to 1000 millimeters per second when it hit the hard stop. This should be compared to the normal velocity of 80 millimeters per second used for slews.