When Automation Fails: Six Infrastructure Incidents

Six investigated incidents from 1999 to 2025, five of them in energy: what each system did or failed to do, what operators saw, and what investigators found.

When automation fails in critical infrastructure, people lose something they were depending on: the ability to command a system, a warning that conditions have changed, or a true reading of what the equipment is doing. The six incidents below show each of those losses. A pipeline control system stopped responding during a pressure surge. A utility's alarm software stalled and told no one. Reservoir level sensors shifted out of position. A train detection circuit reported an occupied track as empty. Gas regulators opened fully because they were sensing a pipe that had been taken out of service. Generation in Spain began disconnecting, and within seconds a cascade of overvoltage disconnections had taken down the Iberian grid.

The investigators in these cases looked beyond the failed equipment, to decisions about testing, maintenance, configuration, and procedure. Deliberate cyberattacks are a separate subject and are left out.

1999: Bellingham Pipeline Rupture

Olympic Pipe Line Company ran a 400-mile pipeline system from Renton, Washington, through a supervisory control and data acquisition (SCADA) system, the software that displays field readings and commands remote pumps and valves. It ran on two identically configured computers, one primary and one backup, and changes to one were mirrored to the other.

Shortly before 3:00 p.m. on June 10, 1999, a controller temporarily assigned as system administrator added new records to the SCADA historical database while the system was operating the pipeline. Database errors began appearing about 3:10. At 3:18:58 a command to start a pump at the unattended Woodinville station failed to execute, and the system stopped responding. Pressure built, protective devices shut down pumps, and an inlet block valve at the Bayview terminal closed. About 3:28 the 16-inch pipeline ruptured at a section weakened by earlier construction damage, releasing about 237,000 gallons of gasoline into a creek in Bellingham.

The National Transportation Safety Board (NTSB) could not establish how much data reached the controllers during the outage, and it found that none of the readily available information showed a rupture. The Bayview valve had closed unexpectedly 41 times in six months, and the line had restarted without incident each time. The line was restarted at 4:11 with a supervisor's approval, which the NTSB said significantly increased the release. The block valves were recorded closed at 4:34, about 1 hour and 5 minutes after the rupture.

About an hour and a half after the rupture, the gasoline ignited and burned about 1.5 miles along the creek. Two 10-year-old boys and an 18-year-old man died, and eight more injuries were documented. The NTSB listed five items in its probable cause. Four concerned the pipe damage, Olympic's inadequate oversight of that work and its misreading of later inspection results, and untested safety devices and a repeatedly closing valve at Bayview. The fifth was Olympic's practice of doing database development on the SCADA system while it was in use, which led to the system becoming non-responsive at a critical time. The NTSB concluded the degraded performance likely came from that work, though testing could not reproduce it, and that a responsive system would probably have let the controller prevent the pressure increase.

2003: Northeast Blackout

The August 14, 2003 blackout affected an estimated 50 million people and 61,800 megawatts of load in eight U.S. states and Ontario. The U.S.-Canada Power System Outage Task Force named four groups of causes: FirstEnergy (FE) and its reliability council did not understand the inadequacies of FE's system; FE had inadequate situational awareness; FE did not adequately manage tree growth along its lines; and reliability organizations did not provide effective real-time diagnostic support. It called the control room computer failures and the line contacts with trees the most important causes.

FE's operators worked from an energy management system (EMS), the control room software that gathers field data and raises alarms. Its alarm and logging software failed shortly after 14:14 Eastern Daylight Time. After that, consoles received no alarms and nothing was printed or logged. The EMS had no function to warn anyone that its alarm system had failed. The rest of the system kept collecting valid data.

FE's analysis suggested the alarm process stalled on one event while incoming data overflowed its buffers. At 14:41 the server hosting it failed and its software moved to the backup as programmed. The stalled application moved too, and the backup failed at 14:54. Strip charts flat-lined, and the Task Force found no indication anyone noticed. Automatic pages reached FE's information technology staff, who did not tell the control room. They restarted a server at 15:08 and saw healthy diagnostics, but the alarm function stayed frozen. A control room operator first told them of the alarm problem at 15:42.

The operators, seeing no alarms, believed their system was satisfactory, and they were surprised by calls from neighboring organizations and field staff that conflicted with that picture. An analyst at the Midwest Independent System Operator (MISO) had turned off the automatic trigger for its state estimator, which estimates grid conditions from real-time measurements and a system model, and forgot to re-enable it. After the Sammis-Star line tripped at 16:05:57, the blackout spread within seven minutes. Many key lines tripped on zone 3 relays responding to overloads with no true fault present. By 16:13, more than 508 generating units at 265 power plants were lost.

2005: Taum Sauk Reservoir Breach

Taum Sauk, a 450-megawatt pumped storage plant in Missouri operated by AmerenUE, pumped water to a mountaintop reservoir and released it to generate power. Operators at a remote control center ran the units. Two systems, both installed in 2004, protected against overpumping. Pressure transducers fed a programmable logic controller (PLC) that shut the pumps off at set levels. Backup conductivity probes near the top of the wall were meant to stop the pumps and sound an alarm.

The transducers hung inside plastic pipes that were not anchored to the concrete face, because of a decision against drilling through a new liner. On September 27, 2005, with the controls reading Elevation 1596, staff found water within four inches of the wall top and adjusted the readings. By October 7, staff had seen that a protective pipe had moved, recognized that levels were reading low, and lowered the shutdown setting.

On the morning of December 14, 2005, an operator manually stopped the last pump at 5:15, on instructions to stop just short of automatic shutoff. The display read Elevation 1593.7. The panel estimated the actual peak at about Elevation 1597.7, roughly four feet higher, and the lowest point of the wall was at 1597. The backup probes produced no shutdown and no alarm. Asked whether they had ever seen unusual indications on the graphical displays, all but one interviewed operator said no. The embankment breached, and 4,350 acre-feet of water drained in about 25 minutes through a state park and campground. Ameren later said a family of five was injured and that all recovered.

The Independent Panel of Consultants convened by the Federal Energy Regulatory Commission (FERC) gave overtopping as the cause. Its primary root causes included the transducers coming loose from their supports and the backup probes being set too high. The top probe sat at Elevation 1597.7, above the tops of 33 wall panels, and logic added in December 2004 required both probes to stay wet for 60 seconds. The panel also cited a one-foot operating margin, almost no visual monitoring of water levels, and the lack of a spillway, which it called a most important root cause.

2009: Washington Metro Collision

This case comes from rail. It is included because the NTSB's report shows a fail-safe control design failing in an unsafe direction. Washington Metrorail used automatic train control to keep trains apart. Track circuits detected trains electrically, and the system sent following trains speed commands, down to 0 mph, based on what the circuits reported. In automatic mode, the system controlled train movement and the operator monitored.

On June 22, 2009, near the Fort Totten station, train 214 stopped inside track circuit B2-304. The NTSB found that parasitic oscillation in the circuit's modules was creating a spurious signal that mimicked a valid one, so the circuit read as vacant with a train on it. At 4:57:20 the system resumed sending 55 mph commands to train 112 behind it, which was in automatic mode. Its operator activated emergency braking, but the NTSB found there was not enough time once train 214 came into full view. The trains collided about 4:58 p.m. Nine people aboard train 112, including the operator, were killed, and emergency agencies reported taking 52 people to hospitals.

The control center gave no usable warning. Its software generated about 3,000 failed-vacant track circuit alarms a week and about 5,000 failed-occupied alarms. These were classed as minor, and unacknowledged minor alarms cleared after 60 seconds. It also deleted more than 1,000 ghost trains a week, false train images caused by malfunctioning circuits. At 4:57:38 that logic dropped train 214's tag, and a second later the train was gone from the display. The NTSB concluded controllers could not have been expected to know a collision was coming. The circuit had missed almost every train for five days, and train operators did not report it because brief losses of speed commands were common.

The NTSB determined the probable cause was the failure of track circuit modules built by GRS/Alstom Signaling, which caused the system to lose detection of train 214 and send commands to train 112, together with the failure of the Washington Metropolitan Area Transit Authority (WMATA) to ensure that a test developed after two 2005 near-collisions at Rosslyn was used systemwide. Contributing factors included WMATA's lack of a safety culture, its failure to maintain and monitor the control system, the manufacturer's failure to provide a maintenance plan, and ineffective oversight. Archived data showed the Rosslyn circuit had the same oscillation problem as far back as 1988.

2018: Merrimack Valley Gas Overpressure

Columbia Gas of Massachusetts supplied a low-pressure gas system in the Merrimack Valley through 14 regulator stations. Each station had a worker regulator and a backup monitor regulator in series, each with a sensing line feeding back pressure from the main. The NTSB called this a redundant closed-loop control system. It cut pressure from about 75 pounds per square inch to about 0.5.

On September 13, 2018, a contractor crew on a main replacement project isolated an old cast iron main. The sensing lines at the Winthrop Avenue station, about half a mile away, were still connected to it. As pressure fell in the abandoned main, the regulators opened further to raise it, and then opened fully, because they no longer sensed system pressure. High-pressure gas entered the low-pressure system. The NTSB found that the condition and operability of the regulators were not a factor.

A monitoring center in Columbus, Ohio, watched the system through SCADA, which could read pressures but had no remote control. It recorded the rise at 3:57 p.m., and the first high-pressure alarm came at 4:04. The controller called an on-call technician at 4:06. By 4:16 the technician was reporting smoke and explosions. About 4:30 a technician at Winthrop Avenue heard the gas flow and isolated the regulators. Field crews closed valves across the system, and closure was confirmed at 7:24 p.m.

The fires and explosions damaged 131 structures, including at least 5 homes that were destroyed. An 18-year-old man was killed, 22 people were taken to hospitals, and 10,894 customers had service shut off. The NTSB determined the probable cause was Columbia Gas of Massachusetts' weak engineering management, which did not adequately plan, review, sequence, and oversee the project that abandoned the main without first relocating the sensing lines. A contributing factor was a system designed and operated without adequate overpressure protection. The NTSB also concluded, citing accidents over 50 years, that systems relying only on sensing lines and regulators are not optimal for preventing overpressurization.

2025: Iberian Blackout

At 12:33 Central European Summer Time on April 28, 2025, continental Spain and Portugal lost power. The European Network of Transmission System Operators for Electricity (ENTSO-E) calls it the most severe blackout on the European system in over 20 years and the first of its kind, an overvoltage blackout. A 49-member expert panel published a factual report in October 2025 and a final report on March 20, 2026. These are final technical findings. They assign no liability, which the panel leaves to national authorities.

Measures that successfully damped oscillations also raised voltage, though it stayed within range. Then generation disconnected in Spain: 355 megawatts at 12:32:57, 727 at 12:33:16, and 928 at 12:33:17. The panel describes a cascade of overvoltage disconnections. By 12:33:19 the peninsula had lost synchronism with the rest of Europe. Automatic load shedding activated in both countries but could not stop the blackout, which the panel attributes to its overvoltage nature.

The panel found no single root cause. Several findings concern control. Renewable generators with a fixed power factor did not adjust reactive power in response to voltage. Voltage control support from conventional generators was not effective, and the requirements on them had no dynamic specification and no economic consequence for missing the target. Shunt reactors, used for static voltage control, were switched manually. At 12:32 about 58 percent of that capacity was connected, and the rest could not be connected in time. Many protection settings diverged from requirements or did not match system needs. The recommendations include modernizing system defense plans to handle fast voltage variations.

What the Cases Have in Common

Each investigation found several causes. The control failure was one part of a longer chain that included pipe damage, overgrown trees, an unanchored instrument, an omitted work order, and a narrow margin between voltage limits and generator disconnection.

Some failures were silent. FE's alarm system could not report its own failure. Taum Sauk's display read about four feet below the real level. The Metrorail circuit read vacant, and the control center software removed the train image. The displays gave no direct indication of the fault, though the picture differed by case. FE's operators did not recognize the clues in calls that conflicted with their screens. Taum Sauk's staff had known since October that the readings ran low. The NTSB concluded that Metrorail's controllers could not have been expected to know a collision was coming.

Backups often gave less protection than they appeared to. Olympic's database changes mirrored to the backup computer. FE's failover carried the stalled application to the second server. Merrimack Valley's worker and monitor regulators sensed the same abandoned main. Taum Sauk's backup had its own flaw: the probes sat above the lowest part of the wall.

Planned work set up several of these failures. A database update, a controls upgrade, an impedance bond replacement, a main replacement, and a troubleshooting session at MISO each changed something in a live system, and nothing in the process around that work caught the consequence. At Metrorail, the change exposed a latent module fault that verification tests failed to reveal. Warnings had also come earlier: 41 valve closures at Bayview, earlier alarm software failures that FE staff described to investigators, water four inches from the top of the wall at Taum Sauk, and two near-collisions at Rosslyn.

Manual fallback was slow or missing when it was needed. Woodinville was unattended. Merrimack Valley's controllers could watch pressures but had no remote control. A technician isolated the Winthrop Avenue regulators about 26 minutes after the first alarm, and closing valves across the system took until 7:24 p.m. The rest of Spain's shunt reactors could not be switched in time. The reports describe systems that people had come to rely on, and that left them little time or information once they stopped working as expected.