3 ms·

> Also, set up monitoring and notification on everything you possibly can. Notifying about everything is a very, very bad idea. You'll drown under notification

by dozzie 12y ago

> Also, set up monitoring and notification on everything you possibly can. Notifying about everything is a very, very bad idea. You'll drown under notifications that do not matter and you will certainly miss those that do matter. There should be as little notifications as possible.

smt88 12y ago
A good rule of thumb is that you should monitor everything you can (tempered with an understanding of performance requirements) and notify for anything that requires action, even if it's not an emergency. It's easy to categorize your notifications after you've received them.
falcolas 12y ago
I used to believe this. I was aggravated by the pager, and would happily acknowledge problems in Nagios just to make the damned thing shut up. More and more pages would come in, the volume increasing each day as "non-issues" piled up. The third time this behavior caused downtime for my clients, I wised up and took my own advice of "uptime is king". I took the "every alert is actionable" to heart, and took a few moments to realize that the action can also be against the monitoring. Alert for disk at 80% on a 2TB volume? Action: Verify growth in Graphite isn't out of the ordinary, and increase the warning threshold to give you about 3-4 weeks of notice that you might need to get bigger disks. Alert for an excessive number of 404 responses? Browse the nginx logs and identify someone trying to hack your corporate-mandated WordPress install. Verify they aren't making any traction, and add exceptions to your 404's so you don't alert on known (and non-whitelisted) endpoints they're hitting. Alert for memory at 80%? What's consuming it; do I have a memory leak and need to restart something? If all is well, and MySQL is just being greedy, up the alert to 90%. Disk capacity warning at 3am? Put some hours around the warning notification, and add a separate "things are growing out of control" alert which doesn't have hours. API endpoint is not responding again, but it's not the system at fault? Add the API developer to the notifications for that alert and remove yourself for a week or two. After an admittedly harrying week of this, you gain two things. One: an operational understanding of what is going on in your system. Two: an alerting system which is tuned to your use case, and which lets you know when you have real problems. Remember - uptime is king.
bbrazil 12y ago
I'd recommend My Philosophy on Alerting by Rob Ewaschuk - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... as a good approach to take. We've fleshed this out a bit more in the Prometheus best practices at http://prometheus.io/docs/practices/alerting/ http://prometheus.io/docs/practices/alerting/ Taking this approach at our company greatly reduced the alert count and improved responsiveness with no degradation in service.

by dozzie 12y ago

smt88 12y ago

A good rule of thumb is that you should monitor everything you can (tempered with an understanding of performance requirements) and notify for anything that requires action, even if it's not an emergency. It's easy to categorize your notifications after you've received them.

falcolas 12y ago

I used to believe this. I was aggravated by the pager, and would happily acknowledge problems in Nagios just to make the damned thing shut up. More and more pages would come in, the volume increasing each day as "non-issues" piled up. The third time this behavior caused downtime for my clients, I wised up and took my own advice of "uptime is king". I took the "every alert is actionable" to heart, and took a few moments to realize that the action can also be against the monitoring. Alert for disk at 80% on a 2TB volume? Action: Verify growth in Graphite isn't out of the ordinary, and increase the warning threshold to give you about 3-4 weeks of notice that you might need to get bigger disks. Alert for an excessive number of 404 responses? Browse the nginx logs and identify someone trying to hack your corporate-mandated WordPress install. Verify they aren't making any traction, and add exceptions to your 404's so you don't alert on known (and non-whitelisted) endpoints they're hitting. Alert for memory at 80%? What's consuming it; do I have a memory leak and need to restart something? If all is well, and MySQL is just being greedy, up the alert to 90%. Disk capacity warning at 3am? Put some hours around the warning notification, and add a separate "things are growing out of control" alert which doesn't have hours. API endpoint is not responding again, but it's not the system at fault? Add the API developer to the notifications for that alert and remove yourself for a week or two. After an admittedly harrying week of this, you gain two things. One: an operational understanding of what is going on in your system. Two: an alerting system which is tuned to your use case, and which lets you know when you have real problems. Remember - uptime is king.

bbrazil 12y ago

I'd recommend My Philosophy on Alerting by Rob Ewaschuk - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... as a good approach to take. We've fleshed this out a bit more in the Prometheus best practices at http://prometheus.io/docs/practices/alerting/ http://prometheus.io/docs/practices/alerting/ Taking this approach at our company greatly reduced the alert count and improved responsiveness with no degradation in service.