4 ms·
> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more probl
by falcolas 12y ago
> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring.
Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict:
- Pages should be [...] actionable
- Symptoms should be monitored, not causes
The problem is that can't act on symptoms, only research them and then act on the causes. If you get an alert that says the DB is down, that's an actionable page - start the DB back up. Whereas, being paged that the connections to the DB are failing is far less actionable - you have to spend precious downtime researching the actual cause first. It could be the network, it could be an intermediary proxy, or it could be the DB itself.
Now granted, if you're only catching causes, there is the possibility you might miss something with your monitoring, and if you double up on your monitoring (that is, checking symptoms as well as causes), you could get noise. That said, most monitoring solutions (such as Nagios) include dependency chains, so you get alerted on the cause, and the symptom is silenced while the cause is in an error condition. And if you missed a cause, you still get the symptom alert and can fill your monitoring gaps from there.
Leave your research for the RCA and following development to prevent future downtime. When stuff is down, a SA's job is to get it back up.
- wereHamster 12y ago> The problem is that can't act on symptoms, only research them and then act on the causes. Have a look at the example at the top of page 5.
- falcolas 12y agoI did. And I see the words "A human just has to parse through a few lines of text ...". Why is a human doing this? Why isn't the tool identifying the cause through a pre-determined dependency tree and only alerting on the salient points instead of relying on a human intelligently parsing through this data when they've been woken up by a page at 3am? My experience has shown me that this kind of parsing can be avoided by using the right tools.
- wereHamster 12y agoOf course if the system can automatically detect the problem and fix it then it should (this is also mentioned somewhere in the document). But at some point it will have to bail and then a human has to have a look.
- robewaschuk 12y agoI don't think there's a contradiction there. "Research them and act on the causes" is _exactly_ the action you should take on a symptom-based alert. Yes, the ultimate action is always to try to address and eliminate the "rootest root cause", but if you start from the symptom you will potentially see more ways to mitigate it then if you start from "database server disk is bad."
- falcolas 12y agoPerhaps not the best choice of problem to use as an example - a Database server disk is bad problem has one real solution, and will cause a whole flood of seemingly unrelated problems. Knowing immediately that it's the server disk (more realistically a raid array going into recovery mode) will save you a lot of time and effort troubleshooting what would appear as a sporadic slow response issue. There's dozens of potential causes for poor response times, of which a raid array in recovery mode is just one. And once you know it's a raid array in recovery mode, you can then take immediate action, something you can't do if you are still busy troubleshooting a sporadic response slowness issue. I feel that ultimately, there's no problem with monitoring for high level symptoms, but they should not be the goal state of monitoring. The goal should be to monitor all possible causes of problems to limit the troubleshooting the SA has to do at 3am when woken by a page. Plus, you should be using a tool which properly silences high level symptoms if there's a problem with a system which is clearly identified as a parent. That is to say, a server being down will silence "db is not responding" alerts.
- derefr 12y agoI think this might be better understood in terms of "workarounds" rather than "solutions", because while "solutions" are important in the long term, "workarounds" are what get users interacting with your app again. Symptom-based alerting lets you apply symptom-based workarounds much sooner than cause-based alerting will let you solve the problem. If your DB's disk is bad, your real problem isn't that the disk is bad; your real problem is that, for example, customers can't buy products from your site. Fixing the symptoms means making your customers able to buy products from your site, not replacing the DB's disk. If you had, say, a failover slave DB, the point of the alert is to tell you to activate the failover process. Replacing the disk is important, but not urgent in the same way activating the failover is. (Note the interesting fact that all alerts will then end up being for things the system could do something in response to on its own. Failing over to a slave can be automatic. Alerts are, in effect, the system saying "I need a human to come help me stop this from happening, because I don't know how to stop it myself.")
- breischl 12y agoI don't know about that, frequently the "cause" that you're alerting on is also a symptom. "DB down" is causing your webapp to fail, but why is the DB down? Misconfiguration, hardware failure, power outage? Heck, maybe the DB is not actually down but there was a network failure that made it unreachable from the monitoring server. My point is that alerting on a "cause" may not actually get you to the root cause, and maybe not even all that much closer than a symtpom.
- falcolas 12y ago> Misconfiguration, hardware failure, power outage? Heck, maybe the DB is not actually down but there was a network failure that made it unreachable from the monitoring server. Sorry, I should have been more specific. DB daemon being down is a cause, and should be monitored. Hardware down is a cause, and should be monitored. Network availability is a cause and should be monitored. Power outage... you get my drift. I think the root cause of my disagreement with this document is the lack of a proper dependency tree in the alerting tool. Their tool appears to want to alert on any and every monitored problem, which necessitates limiting what you monitor for fear of a pager flood. A proper tool can address this problem correctly.
- rhizome 12y ago911: PING TO GOOGLE >100ms
- jrochkind1 12y agoYep, I thought of that when I was reading it too, but he also talked about having a 'dashboard' system that lets engineers easily determine causes for symptoms. His argument is _heavily_ dependent on having such a system -- and his argument about symptoms-not-causes _only_ about _alerts_ (specifically pages that go directly and immediately to engineers). In that context, I find it persuasive based on my own experience. (And I wish I had the kind of dashboard he described available; I'd love to see a presentation or blog post about _that_!)
- ithkuil 12y ago"Every page should require intelligence to deal with: no robotic, scriptable responses." This omits the implicit "because robotic, scriptable responses should be dealt by robots and scripts". You should have monitoring for DB is down. But you should page on "product is not working". Having a monitoring system that lets you quickly find what's wrong is extremely important, but you shouldn't be woken up / or distracted for unactionable alerts.
- gfodor 12y agocan't upvote enough. not every monitoring check has to page. this is the key insight -- more visibility is always better, more alerting is not.
- fh973 12y agoOn he topic cause vs symptom-based alerting there is one aspect missing: decoupling. The systems that Rob is responsible for are decoupled from individual pieces of hardware (redundancy/fault-tolerance) and degrade gracefully (if one part fails only a fraction of the working set is affected). Otherwise the huge scale would not be manageable as some piece is always breaking. So yes: if you run systems that are not somewhat decoupled from their base, cause==symptom and cause-based alerting is indeed the way to go. And you will be woken up by every single piece breaking. The way to improve this is decoupling, and then you'll also switch to symptom-based alerting.