4 ms·
Conflating the Roles of Alerting and Dashboards
- mhb 11y agoWhat's the distinction between a "contradistinction" and a distinction?
- Terr_ 11y agoNot an English-major, but I suspect it goes something like: (D) "Here's a truck. Also, it's a BLUE truck." (CD) "Here's a truck. Also, it's a BLUE truck, not a RED one." But on a practical level I doubt anybody cares about this contradistinction in terms. :P
- morninj 11y agos/English-major/English major/g —an English major
- KyleBrandt 11y ago6 bytes ;)
- neogodless 11y agoI'm having trouble making any sort of contradistinction between the two.
- neogodless 11y agoAh - one of these words communicates the meaning behind it. The other distracts the reader from the original discussion.
- Terr_ 11y agoMistake 3: Putting different people in charge of alert-setting than alert-responding In the best case, it increases the risk of alarm-fatigue. In the pathological cases, it means the technical system becomes a cross-group battleground for shifting blame and workload.
- coda_ 11y agoHah! That's a very good point. I can think of several instances I've been involved in where the person who decided what triggers the alarm never actually had to respond to the alarm. I guess it's relevant for anything you are building... not just alarms. The end users of the software need to be involved in the building of it.
- Terr_ 11y agoI think it's a matter of that most elusively difficult-to-engineer qualities: Trust. (Making sure the two people are in the same team are just an easier way to get it.) The setter trusts that they don't have to micromanage or over-stimulate responders to react and fix the problems. The person dealing with the alarm trusts that the setter isn't crying-wolf or shirking their own duties to minimize alerts. > The end users of the software need to be involved in the building of it. That's my personal bugbear at work right now. My team was literally (albeit indirectly) told to stop talking to users and to route everything through the same "proper channels" which we had written off as clueless. I'm gradually working my way free of "maybe they'll change this time" narrative that's kept me there since my original employer was acquired and merged into the blob.
- ChrisArgyle 11y agoI've seen this go badly even when the first responders are the ones configuring the alerts. For example those getting paged for slow app response times want DBAs to be paged for slow query performance who in turn want System Admins to be paged for high disk latency, etc. These are all reasonable requests but weren't always implemented due to poor communication and/or team relationships.
- dredmorbius 11y agoGood, though I find Rob Ewashuk's "My Philosophy on Alerting" essay (Google SRE) better: https://docs.google.com/a/gravitant.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/preview?sle=true&pli=1#heading=h.fs3knmjt7fjy https://docs.google.com/a/gravitant.com/document/d/199PqyG3U... Particularly: Pages [alerts] should be urgent, important, actionable, and real. They should represent either ongoing or imminent problems with your service. Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. You should almost always be able to classify the problem into one of: availability & basic functionality; latency; correctness (completeness, freshness and durability of data); and feature-specific problems. Symptoms are a better way to capture more problems more comprehensively and robustly with less effort. Include cause-based information in symptom-based pages or on dashboards, but avoid alerting directly on causes. The further up your serving stack you go, the more distinct problems you catch in a single rule. But don't go so far you can't sufficiently distinguish what's going on. If you want a quiet oncall rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical.