5 ms·

I think the only cases of major system failures that could be blamed on human error, are ones where people INTENTIONALLY did wrong stuff, sadly, these cases do

I think the only cases of major system failures that could be blamed on human error, are ones where people INTENTIONALLY did wrong stuff, sadly, these cases do exist and were very lethal. 1. First nuclear accident: the theory is one operator was upset that another operator was screwing his wife, and removed the control rods 100% when he was scheduled to work on them, causing a sudden spike in temperature and then a water hammer that killed everyone in the building. (and made it JUMP!!!) 2. Chernobyl, although many people (specially anti-nuclear) blame the accident on the design, it was obvious deliberate error, even if the intentions weren't malicious, the operators ignored alarms, then DISABLED the alarms, and intentionally pushed the reactor past safety limits, even if it didn't had the design flaw it had, it was possible they would still find a way to blow it up if they kept their behaviour. 3. Depressed airline pilot, locked co-pilot out of the cockpit and crashed on the ground on purpose. There are probably other situations out there... thing is, those are situations no matter how well the system is designed, it will still happen, it is impossible to make a "malicious-proof" system, after all, even a rock, with zero moving parts, can be used in the "wrong way" and kill someone.

MauranKilom 6y ago
More info for 1.: https://en.wikipedia.org/wiki/SL-1 https://en.wikipedia.org/wiki/SL-1
DevX101 6y ago
There's a good argument that these are still system failures. If a single individual can intentionally blow up a nuclear reactor, that's a massive point of failure. Humans are often irrational and prone to anger, depression, malice, and greed. Any of those impulses could be motivation for destructive action. Nuclear submarines avoid this by requiring multiple personnel to simultaneously coordinate in activating a launch. On Chernobyl, my take away from the excellent Netflix mini-series was that there was a massive cultural failure, where everyone abdicated responsibility, and there no ability for employees to counter poor decisions from superiors. Culture and organizational design are also important components of complex systems
asoneth 6y ago
I don't have any insight into why the Chernobyl's operators disabled the alarms, but there is a very common effect known as "alarm fatigue" [1] whereby poorly-tuned alarm thresholds are ignored or actively sabotaged by operators so that they can perform their regular work. For legal & CYA reasons system designers often set alarm thresholds so low that false alarms are extremely common. When operators naturally start to tune out the constant background alarms then the designer can either fix the issue by making better alarm thresholds or they can make the alarms more insistent which sets off an arms race for operator attention. If you're interested in specific stories I'd recommend "Set Phases on Stun" by Steven Casey. (It shows up in the syllabus of a number of Human-Computer Interaction classes.) But as a general heuristic: If one operator has trouble dealing with the number of false alarms you can just hire a new operator, but if most of the operators find it problematic then the problem clearly isn't with the operators. 1: https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue
- laurent92 6y ago
  > Alarm fatigue The notable example of alarm fatigue is the designated pillow on board of the Tu-144 (the Russian copy of the Concorde) to stuff into the alarm horn. The wikipedia page is itself a “bestof” bad engineering/operating practices, which are worth a good laugh now that it is not flying anymore. See chapter “Reasons for cancellation” - https://en.wikipedia.org/wiki/Tupolev_Tu-144 https://en.wikipedia.org/wiki/Tupolev_Tu-144
- dreamcompiler 6y ago
  Alarm fatigue might have played a minor role in Chernobyl but the main cause was authoritarian management trying to meet a scheduled deadline to run a test that the reactor was incapable of running safely in its current operational state. They purposely ignored the alarms in order to run the test. There was also a design flaw nobody knew about (positive void coefficient) that caused the scram make things worse, but they'd never have needed to scram if the deputy chief engineer on duty had cared more about safety than meeting his milestone.
  - hinkley 6y ago
ekimekim 6y ago
For 1, it should be noted that that is only one theory. Another less lurid theory is that while attempting to pull the rod a small distance out as part of a routine maintenance procedure, the rod stuck. The operator gave it a yank to unstick it and accidentially withdrew it the whole way.
bityard 6y ago
> the theory is one operator was upset that another operator was screwing his wife That's _one_ theory and it's not even the most likely one, it's just the only one they like to trot out whenever a History Channel producer needs to fill time between ads. The control rods in SL-1 were prone to getting stuck in their channels for a variety of reasons. They were apparently not light either, as part of the investigation for the accident, they built a mockup control rod assembly that was weighted at 84 pounds. For military-age adult male, this is right in the range being able to move, without a lot of control over how _much_ it is moved. It is, however, clearly a design failure that the removal of a single rod allowed the reactor to go critical, and/or that the rod was able to be removed by a human to the extent it was.
lawrjone 6y ago
There's an excellent talk from Nick Means (GitHub) called "Who Destroyed Three Mile Island?". Link: https://www.youtube.com/watch?v=1xQeXOz0Ncs https://www.youtube.com/watch?v=1xQeXOz0Ncs One thing it discusses is how the alerts inside the control hub were so noisy, and were logged via a reasonably slow printer, that the alerts which suggested the reactor was in serious trouble only printed hours after you could do anything. And the alert light that said "shits fucked" was placed right next to the light about an elevator being out of order. Human error is rarely intentional, and even if it is, a system that fails in response has missed the mark. The talk explains the amazing finds made by the government committee hired to investigate the incident, once they left the idea of 'human error' at the door. I do suggest you watch the video, but as a teaser: it turned out that at the time, most US nuclear engineers had been sourced from nuclear subs. Subs have totally different rules on how to manage the reactor, and train their staff inappropriately for a land-based reactor. Pretty significant cultural find...

MauranKilom 6y ago

More info for 1.: https://en.wikipedia.org/wiki/SL-1 https://en.wikipedia.org/wiki/SL-1

DevX101 6y ago

There's a good argument that these are still system failures. If a single individual can intentionally blow up a nuclear reactor, that's a massive point of failure. Humans are often irrational and prone to anger, depression, malice, and greed. Any of those impulses could be motivation for destructive action. Nuclear submarines avoid this by requiring multiple personnel to simultaneously coordinate in activating a launch. On Chernobyl, my take away from the excellent Netflix mini-series was that there was a massive cultural failure, where everyone abdicated responsibility, and there no ability for employees to counter poor decisions from superiors. Culture and organizational design are also important components of complex systems

asoneth 6y ago

I don't have any insight into why the Chernobyl's operators disabled the alarms, but there is a very common effect known as "alarm fatigue" [1] whereby poorly-tuned alarm thresholds are ignored or actively sabotaged by operators so that they can perform their regular work. For legal & CYA reasons system designers often set alarm thresholds so low that false alarms are extremely common. When operators naturally start to tune out the constant background alarms then the designer can either fix the issue by making better alarm thresholds or they can make the alarms more insistent which sets off an arms race for operator attention. If you're interested in specific stories I'd recommend "Set Phases on Stun" by Steven Casey. (It shows up in the syllabus of a number of Human-Computer Interaction classes.) But as a general heuristic: If one operator has trouble dealing with the number of false alarms you can just hire a new operator, but if most of the operators find it problematic then the problem clearly isn't with the operators. 1: https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue

laurent92 6y ago
> Alarm fatigue The notable example of alarm fatigue is the designated pillow on board of the Tu-144 (the Russian copy of the Concorde) to stuff into the alarm horn. The wikipedia page is itself a “bestof” bad engineering/operating practices, which are worth a good laugh now that it is not flying anymore. See chapter “Reasons for cancellation” - https://en.wikipedia.org/wiki/Tupolev_Tu-144 https://en.wikipedia.org/wiki/Tupolev_Tu-144
dreamcompiler 6y ago
Alarm fatigue might have played a minor role in Chernobyl but the main cause was authoritarian management trying to meet a scheduled deadline to run a test that the reactor was incapable of running safely in its current operational state. They purposely ignored the alarms in order to run the test. There was also a design flaw nobody knew about (positive void coefficient) that caused the scram make things worse, but they'd never have needed to scram if the deputy chief engineer on duty had cared more about safety than meeting his milestone.
- hinkley 6y ago

ekimekim 6y ago

For 1, it should be noted that that is only one theory. Another less lurid theory is that while attempting to pull the rod a small distance out as part of a routine maintenance procedure, the rod stuck. The operator gave it a yank to unstick it and accidentially withdrew it the whole way.

bityard 6y ago

> the theory is one operator was upset that another operator was screwing his wife That's _one_ theory and it's not even the most likely one, it's just the only one they like to trot out whenever a History Channel producer needs to fill time between ads. The control rods in SL-1 were prone to getting stuck in their channels for a variety of reasons. They were apparently not light either, as part of the investigation for the accident, they built a mockup control rod assembly that was weighted at 84 pounds. For military-age adult male, this is right in the range being able to move, without a lot of control over how _much_ it is moved. It is, however, clearly a design failure that the removal of a single rod allowed the reactor to go critical, and/or that the rod was able to be removed by a human to the extent it was.

lawrjone 6y ago

There's an excellent talk from Nick Means (GitHub) called "Who Destroyed Three Mile Island?". Link: https://www.youtube.com/watch?v=1xQeXOz0Ncs https://www.youtube.com/watch?v=1xQeXOz0Ncs One thing it discusses is how the alerts inside the control hub were so noisy, and were logged via a reasonably slow printer, that the alerts which suggested the reactor was in serious trouble only printed hours after you could do anything. And the alert light that said "shits fucked" was placed right next to the light about an elevator being out of order. Human error is rarely intentional, and even if it is, a system that fails in response has missed the mark. The talk explains the amazing finds made by the government committee hired to investigate the incident, once they left the idea of 'human error' at the door. I do suggest you watch the video, but as a teaser: it turned out that at the time, most US nuclear engineers had been sourced from nuclear subs. Subs have totally different rules on how to manage the reactor, and train their staff inappropriately for a land-based reactor. Pretty significant cultural find...