3 ms·
Mistake 3: Putting different people in charge of alert-setting than alert-responding In the best case, it increases the risk of alarm-fatigue. In the pathologi
by Terr_ 11y ago
Mistake 3: Putting different people in charge of alert-setting than alert-responding
In the best case, it increases the risk of alarm-fatigue. In the pathological cases, it means the technical system becomes a cross-group battleground for shifting blame and workload.
- coda_ 11y agoHah! That's a very good point. I can think of several instances I've been involved in where the person who decided what triggers the alarm never actually had to respond to the alarm. I guess it's relevant for anything you are building... not just alarms. The end users of the software need to be involved in the building of it.
- Terr_ 11y agoI think it's a matter of that most elusively difficult-to-engineer qualities: Trust. (Making sure the two people are in the same team are just an easier way to get it.) The setter trusts that they don't have to micromanage or over-stimulate responders to react and fix the problems. The person dealing with the alarm trusts that the setter isn't crying-wolf or shirking their own duties to minimize alerts. > The end users of the software need to be involved in the building of it. That's my personal bugbear at work right now. My team was literally (albeit indirectly) told to stop talking to users and to route everything through the same "proper channels" which we had written off as clueless. I'm gradually working my way free of "maybe they'll change this time" narrative that's kept me there since my original employer was acquired and merged into the blob.
- ChrisArgyle 11y agoI've seen this go badly even when the first responders are the ones configuring the alerts. For example those getting paged for slow app response times want DBAs to be paged for slow query performance who in turn want System Admins to be paged for high disk latency, etc. These are all reasonable requests but weren't always implemented due to poor communication and/or team relationships.