9 ms·
If the expectation is that the on-call person should fix all the issues that arise during their shift, you either need a very well defined runbook, or can only
by toast0 2y ago
If the expectation is that the on-call person should fix all the issues that arise during their shift, you either need a very well defined runbook, or can only have people on-call who have deep understanding of the whole system.
I guess that's a model. But every runbook I've seen has a clear call to escalate if the conditions don't seem to match.
Sometimes the runbook will have procedures to disable things until the business day, in which case you don't need to page anybody, but the service will be degraded until the responsible party can manage it. If the procedure doesn't work, someone will get paged.
IMHO, the most important part of a runbook is the escalation process. And probably the most important meta task of an on-call rotation is tracking escalations and ensuring they're dealt with.
Norms depend on your business, but if you get a lot of escalations outside of business hours, you either need to fix your stuff so it doesn't need escalation, or you need to staff your stuff so escalation is to people who are in their business hours.
Edit: I'll also add that reducing incident frequency is good, but when it drops from once a quarter to once a year, new hires won't get osmositic training anymore. When it drops from once a year to once every other year, team muscle memory will have atrophied. It's worth doing some periodic training/refreshing when things are running well.
- spike021 2y agoFully agreed with all this. Also, if there's a bottleneck where an oncall needs to rely on a teammate with more experience with the subject matter, then make sure that's noted down in a retro. Hopefully an action item can be made up and completed where said person does some knowledge transfer, at least into a run book or other documentation.
- incidenrly 2y agoIs it worth having psuedo production services that get chaosed monkeyed into another dimension at a random time and pager alerts on them. Effectively: a drill!
- toast0 2y agoI don't think chaos monkey works for incident drills. Anything the monkey can do is going to be easy to detect (probably). You can do some amount of drills with periodic disaster recovery tests --- twice a year do a manual failover of a colo, etc.