4 ms·
One of the links is to Google’s Sight Reliability Workbook (https://sre.google/workbook/table-of-contents/ https://sre.google/workbook/table-of-contents/). At m
by yashap 6y ago
One of the links is to Google’s Sight Reliability Workbook (https://sre.google/workbook/table-of-contents/ https://sre.google/workbook/table-of-contents/). At my last company, we implemented SLOs based on the advice in this workbook, and I thought it was an excellent approach. Having internal reliability targets that directly map to the meaningful parts of the customer experience, and setting up dashboards and alerts based on these targets, is a very powerful approach to achieving reliability that matters for users.
A great doc from Google that they didn’t link to above is “My Philosophy on Alerting”, by Rob Ewaschuk (https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/preview# https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa...). A must read for anyone setting up alerts IMO.
- was_boring 6y agoThanks for sharing the second link. It's the first time seeing it an encapsulates my own philosophy on alerting well, while providing examples of what to alert on. I've been with teams who treat alerts as informational and then quickly get alert fatigue. Then when something actually does happen they don't recognize it because they've trained themselves to ignore all alerts. At my current company we regularly review our alerts to make sure they are actionable and meaningful still.
- yashap 6y agoNo problem! And agreed, the advice on “alert on symptoms users actually experience, and only on things a human actually has to deal with” is great advice. Alerts that don’t fit this should be quickly tweaked or disabled, to avoid alert fatigue. “Something went wrong, but at acceptable levels/without much user impact” belongs in dashboards, not alerts. Worth being able to monitor, but alerts are for urgency, and that’s not urgent.