4 ms·
These are really good tips, but a lot of them could get their own blog post just about why they're so important. Just the "every alert should be actionable by
by 0xbadcafebee 6y ago
These are really good tips, but a lot of them could get their own blog post just about why they're so important.
Just the "every alert should be actionable by somebody at 3AM" requires a lot of work, because the implications of a single alert and how to respond to it can become very deep. And you probably won't meet it until you actually get the call and realize you missed something.
I prefer to imagine alerts as if they're going to your Grandmother. She can turn her computer on and surf the web and make phone calls. But she doesn't have tribal knowledge about your product, or very specific technical knowledge. If she can't figure out what to do from reading the runbook in the alert, it's likely your on-call people can't either and will just end up waking up other people, and at that point you're just delaying resolving the alert and waking up too many people.