5 ms·

"You get 711 alerts, you only see 18 of them, you are like “ehh 18 alerts is pretty much the normal number,” you override them all without reading." I believe

"You get 711 alerts, you only see 18 of them, you are like “ehh 18 alerts is pretty much the normal number,” you override them all without reading." I believe this is called an 'alert that cried wolf'.

Arrath 2y ago
Alarm Fatigue
- pjacotg 2y ago
  There's a 99% visible episode [1] on this very issue. Worth a listen! [1] https://99percentinvisible.org/episode/sound-and-health-hospitals/ https://99percentinvisible.org/episode/sound-and-health-hosp...
blackeyeblitzar 2y ago
This is a phenomenon in so many different jobs, and is called “alarm fatigue”. From https://www.nytimes.com/2010/08/01/weekinreview/01wald.html https://www.nytimes.com/2010/08/01/weekinreview/01wald.html: > On Tuesday, the National Transportation Safety Board said that a crash last year on the Washington subway system that killed nine people had happened partly because train dispatchers had been ignoring 9,000 alarms per week. Air traffic controllers, nuclear plant operators, nurses in intensive-care units and others do the same.
- mgkimsal 2y ago
  and... 'fixing' it can cause its own issues. I've been involved in some systems that routinely spit out dozens of warnings per day. After 'fixing' some of them - maybe getting a system down to a few per week, bug reports started coming in that the system was 'broken'. Because people noticed the boxes and messages they routinely ignored were gone, and this must be a problem. Lots of re-education may need to happen on a large scale to actually 'fix' alarm fatigue across the board.
  - tristor 2y ago
    This triggered a repressed memory. I was once on an ops team responsible for an application that had been turned over to us written in Java. The application ran in Tomcat, and prior to turning it over to us, the external vendor had "helpfully" built some monitoring and alerting for the application. One of the metrics they monitored was how many errors per minute and per hour were happening the application's logs. When they turned this application over to us, they did not properly hand over the monitoring system or even tell us that it existed, and it was sending alerts to another external vendor team which had no association to my ops team. Several years went by... then because we finally had the time our ops team decided to audit and improve this application, including rewriting significant portions of its code as was pretty typical when we took over an application built by an external vendor. Long story short, we resolved almost all of the errors by the expediency of simply correctly interacting with the database behind the application instead of relying on whatever horrific dumpster fire output to the DB that Hibernate was doing. That's when things got super interesting, because the external vendor team that was receiving these monitoring alerts (unbeknownst to my team) had more than 100% turnover in the intervening period of time, having lost all knowledge about this, but suddenly starting getting a bunch of critical urgency alerts going off because the application was no longer emitting the amount of errors expected for the alarm thresholds. This ended up triggering a massive company-wide incident call, even though there was no external customer impact. I'm leaving a lot of details out, but let's just say it sucked up days of my life and was a massive waste of time for all of that. Fun example of fixing something and causing an "issue" because behavior changed from expectations/baseline.
    - maicro 2y ago
      "The errors let you know it's working; it's when the errors stop you have to worry."
      - opwieurposiu 2y ago
        For old cars we say say "If there ain't no oil under it, then there ain't no oil in it."
      - deleted 2y ago
- HL33tibCe7 2y ago
  > nurses in intensive-care units and others do the same. The last time I was in hospital, I came out the next day and my ears were ringing for a couple of days from the _constant_ ringing of alarms.
  - amatecha 2y ago
    Every time I've been in a hospital (visiting people), I have a frequent sensation of wanting to get out of there. I hope/imagine I will never need to be in a hospital, because I know it will be hell for me. I have extremely good hearing, and am very affected by sound. Visiting people in hospital requires serious effort from me to maintain focus on the person and those I'm there with, and ignoring such a shocking degree of noise I can barely believe it's the accepted norm. I can't believe how many devices and processes purposely make noise as a part of _normal operation_, where there's not even anything amiss!
  - blackeyeblitzar 2y ago
    Yes I really hate that at hospitals. I am certain it is bad for health and recovery of patients and unnecessarily increases stress for everyone.
    - nucleardog 2y ago
      I have no doubt about this. My wife was in the hospital because she’d essentially stressed herself to the point of a heart arrhythmia. The irregular rhythm meant the equipment was basically useless at accurately determining her heart rate and the readings fluctuated wildly. So what did she do for several hours? Laid in a bed anxious about the medical concerns, anxious about life, and listened to the monitoring equipment go off multiple times a minute as she crossed various alert thresholds, went back under them, crossed them again, repeat the entire time she was there. I asked the staff to turn the alarms off or at least adjust the thresholds. They just short of flat out said they wouldn’t because if they disabled or adjusted them and anything happened it would be on their head. So my wife spent half a day being told to try and relax while the machine monitoring her heart and respiration and blood pressure constantly screamed at her that everything was wrong.
- ryandrake 2y ago
  I always wondered: Since Alarm Fatigue is well known, why do UX designers and/or product owners keep insisting on adding new alarms for things? Surely we know they increasingly don't work, the more you have.
  - thephyber 2y ago
    Alarms are easy for a single person to add. The better solution is to redesign / engineer systems which automatically solve the alarm condition so no person needs to know about the alarm. That usually requires many people to work together and requires far more complexity to solve, both initially and in ongoing upkeep.
  - bombcar 2y ago
    Because if someone dies because you didn't alarm an alarm, you get yelled at. If someone dies because they ignored an alarm you did alarm, that's on them.
    - nucleardog 2y ago
      There’s some small fraction where someone believes it would be helpful but was mistaken. What you’re describing is definitely the vast majority. When something unexpected or out of the ordinary happens, what do you do about it? - Nothing? Why did you do nothing? This was clearly a problem. This is your fault. - Figure out appropriate error handling? That’s hard work. - Just throw up an alert on everything and make it someone else’s job to sort through it all? Perfect, job done.
  - mrguyorama 2y ago
    Because modern design philosophy has, as a rule, thrown everything we learned before 2010 out the window because that's not what google does. "But the A/B tests show consumers prefer it!". Oh? So you have a degree in stats and experience running scientific tests? No? You're just a programmer who only knows how to enable the A/B test feature of your favorite javascript library? Your A/B tests aren't testing what you think they are testing.
  - BobaFloutist 2y ago
jh00ker 2y ago
Humans are bad at processing large sets of similar data occurring repeatedly. It comes up everywhere: - On-Call alerts received by my engineering team for our microservices that usually self-resolve result in the first action being taken by the engineers to be "just wait and see if it's a false alarm." We work to reduce the number alerts overall to reduce the noise. - I'm reminded of "Cigna saves millions by having its doctors reject claims without reading them" https://news.ycombinator.com/item?id=35304017 https://news.ycombinator.com/item?id=35304017 In order to keep up with the claims, they just auto-rejected them to see if they were appealed in a way to filter out the "noise." - We hear so much about Tesla's "FSD (Supervised)" and it's request that drivers "don't become complacent" but it happens anyway. After enough time behind the wheel with FSD enabled, we are swayed by the string of successes to become overly trusting of the tech.
reportgunner 2y ago
It took me years to find the name "Alarm Fatigue"[0] [0]https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue

Arrath 2y ago

Alarm Fatigue

pjacotg 2y ago
There's a 99% visible episode [1] on this very issue. Worth a listen! [1] https://99percentinvisible.org/episode/sound-and-health-hospitals/ https://99percentinvisible.org/episode/sound-and-health-hosp...

blackeyeblitzar 2y ago

This is a phenomenon in so many different jobs, and is called “alarm fatigue”. From https://www.nytimes.com/2010/08/01/weekinreview/01wald.html https://www.nytimes.com/2010/08/01/weekinreview/01wald.html: > On Tuesday, the National Transportation Safety Board said that a crash last year on the Washington subway system that killed nine people had happened partly because train dispatchers had been ignoring 9,000 alarms per week. Air traffic controllers, nuclear plant operators, nurses in intensive-care units and others do the same.

mgkimsal 2y ago
and... 'fixing' it can cause its own issues. I've been involved in some systems that routinely spit out dozens of warnings per day. After 'fixing' some of them - maybe getting a system down to a few per week, bug reports started coming in that the system was 'broken'. Because people noticed the boxes and messages they routinely ignored were gone, and this must be a problem. Lots of re-education may need to happen on a large scale to actually 'fix' alarm fatigue across the board.
- tristor 2y ago
  This triggered a repressed memory. I was once on an ops team responsible for an application that had been turned over to us written in Java. The application ran in Tomcat, and prior to turning it over to us, the external vendor had "helpfully" built some monitoring and alerting for the application. One of the metrics they monitored was how many errors per minute and per hour were happening the application's logs. When they turned this application over to us, they did not properly hand over the monitoring system or even tell us that it existed, and it was sending alerts to another external vendor team which had no association to my ops team. Several years went by... then because we finally had the time our ops team decided to audit and improve this application, including rewriting significant portions of its code as was pretty typical when we took over an application built by an external vendor. Long story short, we resolved almost all of the errors by the expediency of simply correctly interacting with the database behind the application instead of relying on whatever horrific dumpster fire output to the DB that Hibernate was doing. That's when things got super interesting, because the external vendor team that was receiving these monitoring alerts (unbeknownst to my team) had more than 100% turnover in the intervening period of time, having lost all knowledge about this, but suddenly starting getting a bunch of critical urgency alerts going off because the application was no longer emitting the amount of errors expected for the alarm thresholds. This ended up triggering a massive company-wide incident call, even though there was no external customer impact. I'm leaving a lot of details out, but let's just say it sucked up days of my life and was a massive waste of time for all of that. Fun example of fixing something and causing an "issue" because behavior changed from expectations/baseline.
  - maicro 2y ago
    "The errors let you know it's working; it's when the errors stop you have to worry."
    - opwieurposiu 2y ago
      For old cars we say say "If there ain't no oil under it, then there ain't no oil in it."
    - deleted 2y ago
HL33tibCe7 2y ago
> nurses in intensive-care units and others do the same. The last time I was in hospital, I came out the next day and my ears were ringing for a couple of days from the _constant_ ringing of alarms.
- amatecha 2y ago
  Every time I've been in a hospital (visiting people), I have a frequent sensation of wanting to get out of there. I hope/imagine I will never need to be in a hospital, because I know it will be hell for me. I have extremely good hearing, and am very affected by sound. Visiting people in hospital requires serious effort from me to maintain focus on the person and those I'm there with, and ignoring such a shocking degree of noise I can barely believe it's the accepted norm. I can't believe how many devices and processes purposely make noise as a part of _normal operation_, where there's not even anything amiss!
- blackeyeblitzar 2y ago
  Yes I really hate that at hospitals. I am certain it is bad for health and recovery of patients and unnecessarily increases stress for everyone.
  - nucleardog 2y ago
    I have no doubt about this. My wife was in the hospital because she’d essentially stressed herself to the point of a heart arrhythmia. The irregular rhythm meant the equipment was basically useless at accurately determining her heart rate and the readings fluctuated wildly. So what did she do for several hours? Laid in a bed anxious about the medical concerns, anxious about life, and listened to the monitoring equipment go off multiple times a minute as she crossed various alert thresholds, went back under them, crossed them again, repeat the entire time she was there. I asked the staff to turn the alarms off or at least adjust the thresholds. They just short of flat out said they wouldn’t because if they disabled or adjusted them and anything happened it would be on their head. So my wife spent half a day being told to try and relax while the machine monitoring her heart and respiration and blood pressure constantly screamed at her that everything was wrong.
ryandrake 2y ago
I always wondered: Since Alarm Fatigue is well known, why do UX designers and/or product owners keep insisting on adding new alarms for things? Surely we know they increasingly don't work, the more you have.
- thephyber 2y ago
  Alarms are easy for a single person to add. The better solution is to redesign / engineer systems which automatically solve the alarm condition so no person needs to know about the alarm. That usually requires many people to work together and requires far more complexity to solve, both initially and in ongoing upkeep.
- bombcar 2y ago
  Because if someone dies because you didn't alarm an alarm, you get yelled at. If someone dies because they ignored an alarm you did alarm, that's on them.
  - nucleardog 2y ago
    There’s some small fraction where someone believes it would be helpful but was mistaken. What you’re describing is definitely the vast majority. When something unexpected or out of the ordinary happens, what do you do about it? - Nothing? Why did you do nothing? This was clearly a problem. This is your fault. - Figure out appropriate error handling? That’s hard work. - Just throw up an alert on everything and make it someone else’s job to sort through it all? Perfect, job done.
- mrguyorama 2y ago
  Because modern design philosophy has, as a rule, thrown everything we learned before 2010 out the window because that's not what google does. "But the A/B tests show consumers prefer it!". Oh? So you have a degree in stats and experience running scientific tests? No? You're just a programmer who only knows how to enable the A/B test feature of your favorite javascript library? Your A/B tests aren't testing what you think they are testing.
- BobaFloutist 2y ago

jh00ker 2y ago

Humans are bad at processing large sets of similar data occurring repeatedly. It comes up everywhere: - On-Call alerts received by my engineering team for our microservices that usually self-resolve result in the first action being taken by the engineers to be "just wait and see if it's a false alarm." We work to reduce the number alerts overall to reduce the noise. - I'm reminded of "Cigna saves millions by having its doctors reject claims without reading them" https://news.ycombinator.com/item?id=35304017 https://news.ycombinator.com/item?id=35304017 In order to keep up with the claims, they just auto-rejected them to see if they were appealed in a way to filter out the "noise." - We hear so much about Tesla's "FSD (Supervised)" and it's request that drivers "don't become complacent" but it happens anyway. After enough time behind the wheel with FSD enabled, we are swayed by the string of successes to become overly trusting of the tech.

reportgunner 2y ago

It took me years to find the name "Alarm Fatigue"[0] [0]https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue