I'd recommend My Philosophy on Alerting by Rob Ewaschuk - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit https://docs.goog

I'd recommend My Philosophy on Alerting by Rob Ewaschuk - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... as a good approach to take. We've fleshed this out a bit more in the Prometheus best practices at http://prometheus.io/docs/practices/alerting/ http://prometheus.io/docs/practices/alerting/ Taking this approach at our company greatly reduced the alert count and improved responsiveness with no degradation in service.