4 ms·
> But how do you foster a culture in which everyone keeps it up to date? Every service has a runbook. To ensure that they're kept up to date, teams are require
by echelon 4y ago
> But how do you foster a culture in which everyone keeps it up to date?
Every service has a runbook. To ensure that they're kept up to date, teams are required to do twice-yearly Disaster Recovery (DR) drills where they fail their service out of one region or data center and into another. (This is for each service. They can be rescheduled - modulo some nagging - if absolutely necessary, but you still need to do them eventually as a part of OKRs / KTLO.)
DR exercises by necessity require that the runbook is up to date and that capacity measures and load testing are recent and accurate. By doing these semi-regularly, your team retains full familiarity with the operational characteristics of the system. Your service, your data stores, your dependencies, and your dependents. Hopefully you (or adjacent teams) won't discover any unknowns, but if you do, they also belong in a runbook or postmortem with action items set to address the issues.
Furthermore, every team keeps a weekly "oncall log", which is essentially a human aggregated and summarized log of pages received. The oncall engineer logs anything that crops up, including new failure classes, and triages fixing them (or gets assistance if they're junior). Novel happenings can easily be discussed during oncall handoff or briefly during standup. Some of these items will naturally find their way into the relevant runbooks.
And as another commenter suggested, use Google Docs. These should be super easy to edit with zero friction. The requirement of creating a PR increases the odds that something will get skipped.
FWIW, several of my previous teams required more than five nines of uptime as we'd lose millions for outages. We had a very mature process.