4 ms·

I don't have any insight into why the Chernobyl's operators disabled the alarms, but there is a very common effect known as "alarm fatigue" [1] whereby poorly-t

I don't have any insight into why the Chernobyl's operators disabled the alarms, but there is a very common effect known as "alarm fatigue" [1] whereby poorly-tuned alarm thresholds are ignored or actively sabotaged by operators so that they can perform their regular work. For legal & CYA reasons system designers often set alarm thresholds so low that false alarms are extremely common. When operators naturally start to tune out the constant background alarms then the designer can either fix the issue by making better alarm thresholds or they can make the alarms more insistent which sets off an arms race for operator attention. If you're interested in specific stories I'd recommend "Set Phases on Stun" by Steven Casey. (It shows up in the syllabus of a number of Human-Computer Interaction classes.) But as a general heuristic: If one operator has trouble dealing with the number of false alarms you can just hire a new operator, but if most of the operators find it problematic then the problem clearly isn't with the operators. 1: https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue

laurent92 6y ago
> Alarm fatigue The notable example of alarm fatigue is the designated pillow on board of the Tu-144 (the Russian copy of the Concorde) to stuff into the alarm horn. The wikipedia page is itself a “bestof” bad engineering/operating practices, which are worth a good laugh now that it is not flying anymore. See chapter “Reasons for cancellation” - https://en.wikipedia.org/wiki/Tupolev_Tu-144 https://en.wikipedia.org/wiki/Tupolev_Tu-144
dreamcompiler 6y ago
Alarm fatigue might have played a minor role in Chernobyl but the main cause was authoritarian management trying to meet a scheduled deadline to run a test that the reactor was incapable of running safely in its current operational state. They purposely ignored the alarms in order to run the test. There was also a design flaw nobody knew about (positive void coefficient) that caused the scram make things worse, but they'd never have needed to scram if the deputy chief engineer on duty had cared more about safety than meeting his milestone.
- hinkley 6y ago
  Engineers constantly rail against the “mind over matter” hubris of managers. If at first they are right they just push their luck again and again.
hinkley 6y ago
I rediscovered this effect myself and it’s why I harp on people for flaky tests. Eventually you discover the build has been broken for hours, everyone assumes they’re just the same old errors, and pushes rebuild without even looking at the error messages. Everyone does it. I’ve caught myself doing it. So I move to the previous domino and work on that.
- zimpenfish 6y ago
  Same goes for log output - if you get thousands of lines every day, no-one ever looks at the logs and things get missed. If you only get log messages when things are really broken, people actually notice.
  - Macha 6y ago
    I mean, this is what log levels are for. My personal standard is log level error is for things raising an alert over that you need a human to look at, warning is for stuff that can raise an alert if it goes over a threshold, info is only for looking at when a problem has already been identified, so that you can try understand the lead up to the problem, and debug is incredibly spammy stuff only turned on when the above gives insufficient details to recreate the problem locally.
    - hinkley 6y ago
      Stats are good for things that may be a problem or may not. It’s hard to get a sense of rate with log analysis. You can, but comparing several stats to each other at once is painful.
myself248 6y ago
It blows my mind that this is still a problem; Western Electric was already making allowances for it in what I believe to be the 1930s: Most telephone hardware has some form of self-checking and fault detection. Alarms are classified into "minor" and "major", sometimes also "power" or "critical". Each piece of equipment has its own alarm indicator lamps, and contacts for external alarms as well. Typically the external contacts are wired to both an "aisle light" which aggregates all the equipment in that lineup, and a floor-wide or section-wide "alarm sounder". When the sounder goes off, you look down the aisle to see which lineup has the trouble, then walk down the lineup to find the individual unit and address the problem. But first, you push the ACO button. The alarm cutoff button silences the sounder, which won't come back on unless something new happens. ACO also tells the other workers that someone is on the case. Crucially, it preserves attention. If the sounder was going the whole time someone was working on a repair, nothing would ever get done, because the workers would be down at the corner bar trying to get some sanity back.
- asoneth 6y ago
  In my experience it's sometimes a result of mismatched incentives resulting in a prisoner's dilemma rather than a lack of awareness or engineering ability. The designer responsible for an alarm wants to avoid being blamed for the operator missing that specific alarm so they're incentivized to make their alarm as prominent as they can get away with. Of course once most designers are doing that then they have to keep doing it or risk get drowned out. One way to fix that is by having someone who is responsible for the entire experience who can dictate shared prioritization, requirements, alerting frameworks, rate limiters, etc. In cases where the operator uses multiple products from different companies (e.g. in a hospital) that's especially hard which is why there's a cacophony of noise even though everyone involved acknowledges that it's a problem.
  - myself248 6y ago
    > In cases where the operator uses multiple products from different companies (e.g. in a hospital) that's especially hard Somehow this works just fine in telecom. I was handling equipment from DSC, Alcatel, Fujitsu, Marconi, Rockwell, Cerent, Nortel, Pirelli, CarrierAccess, Tellabs, Cisco, ADC, Lucent, and more. There are Telcordia and NEBS standards for alarm severity and wiring, and everything just works when you hook it up. Does the medical industry not have standards?
dghlsakjg 6y ago
My car dashboards (and most cars as far as I know) are actually a good example of avoiding alarm fatigue. Red lights mean that the sensors have detected a state that may damage the car or put you in danger. You should stop immediately. Examples: low oil pressure, overheat, emergency brake still on, low brake fluid. You can safely tell a new driver that if they are driving and they see a red light on the dash it means stop right now. Yellow means that something is operating outside of nominal ranges and you should try and fix it soon. Examples: low tire pressure, low washer fluid, check engine light.

laurent92 6y ago

> Alarm fatigue The notable example of alarm fatigue is the designated pillow on board of the Tu-144 (the Russian copy of the Concorde) to stuff into the alarm horn. The wikipedia page is itself a “bestof” bad engineering/operating practices, which are worth a good laugh now that it is not flying anymore. See chapter “Reasons for cancellation” - https://en.wikipedia.org/wiki/Tupolev_Tu-144 https://en.wikipedia.org/wiki/Tupolev_Tu-144

dreamcompiler 6y ago

Alarm fatigue might have played a minor role in Chernobyl but the main cause was authoritarian management trying to meet a scheduled deadline to run a test that the reactor was incapable of running safely in its current operational state. They purposely ignored the alarms in order to run the test. There was also a design flaw nobody knew about (positive void coefficient) that caused the scram make things worse, but they'd never have needed to scram if the deputy chief engineer on duty had cared more about safety than meeting his milestone.

hinkley 6y ago
Engineers constantly rail against the “mind over matter” hubris of managers. If at first they are right they just push their luck again and again.

hinkley 6y ago

I rediscovered this effect myself and it’s why I harp on people for flaky tests. Eventually you discover the build has been broken for hours, everyone assumes they’re just the same old errors, and pushes rebuild without even looking at the error messages. Everyone does it. I’ve caught myself doing it. So I move to the previous domino and work on that.

zimpenfish 6y ago
Same goes for log output - if you get thousands of lines every day, no-one ever looks at the logs and things get missed. If you only get log messages when things are really broken, people actually notice.
- Macha 6y ago
  I mean, this is what log levels are for. My personal standard is log level error is for things raising an alert over that you need a human to look at, warning is for stuff that can raise an alert if it goes over a threshold, info is only for looking at when a problem has already been identified, so that you can try understand the lead up to the problem, and debug is incredibly spammy stuff only turned on when the above gives insufficient details to recreate the problem locally.
  - hinkley 6y ago
    Stats are good for things that may be a problem or may not. It’s hard to get a sense of rate with log analysis. You can, but comparing several stats to each other at once is painful.

myself248 6y ago

It blows my mind that this is still a problem; Western Electric was already making allowances for it in what I believe to be the 1930s: Most telephone hardware has some form of self-checking and fault detection. Alarms are classified into "minor" and "major", sometimes also "power" or "critical". Each piece of equipment has its own alarm indicator lamps, and contacts for external alarms as well. Typically the external contacts are wired to both an "aisle light" which aggregates all the equipment in that lineup, and a floor-wide or section-wide "alarm sounder". When the sounder goes off, you look down the aisle to see which lineup has the trouble, then walk down the lineup to find the individual unit and address the problem. But first, you push the ACO button. The alarm cutoff button silences the sounder, which won't come back on unless something new happens. ACO also tells the other workers that someone is on the case. Crucially, it preserves attention. If the sounder was going the whole time someone was working on a repair, nothing would ever get done, because the workers would be down at the corner bar trying to get some sanity back.

asoneth 6y ago
In my experience it's sometimes a result of mismatched incentives resulting in a prisoner's dilemma rather than a lack of awareness or engineering ability. The designer responsible for an alarm wants to avoid being blamed for the operator missing that specific alarm so they're incentivized to make their alarm as prominent as they can get away with. Of course once most designers are doing that then they have to keep doing it or risk get drowned out. One way to fix that is by having someone who is responsible for the entire experience who can dictate shared prioritization, requirements, alerting frameworks, rate limiters, etc. In cases where the operator uses multiple products from different companies (e.g. in a hospital) that's especially hard which is why there's a cacophony of noise even though everyone involved acknowledges that it's a problem.
- myself248 6y ago
  > In cases where the operator uses multiple products from different companies (e.g. in a hospital) that's especially hard Somehow this works just fine in telecom. I was handling equipment from DSC, Alcatel, Fujitsu, Marconi, Rockwell, Cerent, Nortel, Pirelli, CarrierAccess, Tellabs, Cisco, ADC, Lucent, and more. There are Telcordia and NEBS standards for alarm severity and wiring, and everything just works when you hook it up. Does the medical industry not have standards?

dghlsakjg 6y ago

My car dashboards (and most cars as far as I know) are actually a good example of avoiding alarm fatigue. Red lights mean that the sensors have detected a state that may damage the car or put you in danger. You should stop immediately. Examples: low oil pressure, overheat, emergency brake still on, low brake fluid. You can safely tell a new driver that if they are driving and they see a red light on the dash it means stop right now. Yellow means that something is operating outside of nominal ranges and you should try and fix it soon. Examples: low tire pressure, low washer fluid, check engine light.