3 ms·

Reminds me of my manager at a job a few years back. His standard was that any steps an on-call needs to take should be doable while they are drunk. You're wakin

Reminds me of my manager at a job a few years back. His standard was that any steps an on-call needs to take should be doable while they are drunk. You're waking people up in the middle of the night or they could be in who knows what sort of cognitive state. The runbook needs to be dead simple to follow. Mitigate now, fix it all up tomorrow when someone's awake and coherent.

cameronh90 5y ago
If it is simple enough to do when drunk, why does it need to be done at all? Typically for us, when something goes wrong that needs intervention, it's the first time that issue has ever happened and it needs to be debugged and understood. Then we'll put a mitigation in so it hopefully doesn't happen again. In my last place, we had a runbook more like you describe, but I prefer prioritising bug fixes over features as it means I haven't had an out of hours call in a year!
- doktorhladnjak 5y ago
  That was the other half of it though. Once it’s simple to mitigate, automate it! To be fair, this was on a team that had A TON of operational debt. Like you’d get paged 30 times _a day_ when on call. Nobody wanted to move to weekly on call from a daily rotation because it would be a total nightmare, but daily meant the incentive to fix something was low because tomorrow it would be someone else’s problem. Of course, the solution was to get the alerts and underlying software under control, which we did and things got better. Many of the non automated run books that remained were things like “look at these three graphs, if they’re doing x, escalate to team y”. We were somewhat hamstrung by political considerations that prevented us from simply directly paging those teams, or limitations of our alerting system that couldn’t link alerts (“if this other alarm is already going off, you don’t have to do anything because team y is already being paged for the root cause”).

cameronh90 5y ago

If it is simple enough to do when drunk, why does it need to be done at all? Typically for us, when something goes wrong that needs intervention, it's the first time that issue has ever happened and it needs to be debugged and understood. Then we'll put a mitigation in so it hopefully doesn't happen again. In my last place, we had a runbook more like you describe, but I prefer prioritising bug fixes over features as it means I haven't had an out of hours call in a year!

doktorhladnjak 5y ago
That was the other half of it though. Once it’s simple to mitigate, automate it! To be fair, this was on a team that had A TON of operational debt. Like you’d get paged 30 times _a day_ when on call. Nobody wanted to move to weekly on call from a daily rotation because it would be a total nightmare, but daily meant the incentive to fix something was low because tomorrow it would be someone else’s problem. Of course, the solution was to get the alerts and underlying software under control, which we did and things got better. Many of the non automated run books that remained were things like “look at these three graphs, if they’re doing x, escalate to team y”. We were somewhat hamstrung by political considerations that prevented us from simply directly paging those teams, or limitations of our alerting system that couldn’t link alerts (“if this other alarm is already going off, you don’t have to do anything because team y is already being paged for the root cause”).