6 ms·

Runbooks are more of an anti-pattern than anything. A list of commands you should run to accomplish X -> that's a script. Or it should be. If accomplishing a t

Runbooks are more of an anti-pattern than anything. A list of commands you should run to accomplish X -> that's a script. Or it should be. If accomplishing a task takes more than a single step, it means there's insufficient automation or programmability, and that's the problem you should fix, not teaching devs how to be better writers of prose. Code can be tested quickly and relatively easily. Docs can't. Docs should be as minimal as possible: "# Server ## Rebooting - ./scripts/server/reboot.sh". Any docs written by a dev should be run by an editor, who should strip them down to the bare minimum. Your typical readme is inundated with waffle, and the actual meat in it is wrong well over half the time. Automate everything.

mooreds 7y ago
> Automate everything. Whenever I see someone say something like this, I'm always curious about where human judgement and/or investigation comes in. Don't get me wrong, I'm happy to automate what can be automated if it makes sense (that is, the action can be easily turned into a script and the cost to create and maintain the automation is not higher than the value created) but aren't there plenty of areas where some level of human judgement is needed and you can't just automate everything? Isn't that kinda what SREs do? Develop intuition for what can be automated, automate what they can, set up process for what they can't?
- epicgiga 7y ago
  In the context of docs/"runbooks", my point is you're not saving anything by doing that instead of scripting. You're going to pay for it in the long run when the docs quickly drift out of date and then people stop trusting them, and the docs have limited value anyway because they're just going to ask for your help in person instead 90% of the time (because that's 10x more effective). Scripts can have similar problems, but they're rapid and testable, so far easier to maintain their integrity. Where judgment is needed, i.e. unique cases, the runbooks won't help much anyway, and its up to the tech lead to work out why that problem happened once they've fixed it, and then either modify the system, or add a new script to fix that particular new problem if it arises again.
- rtpg 7y ago
  I'm basically of the same opinion as you, but recently read this article[0] on "do nothing" scripts and really like it. The idea is to write at your process _as a script_ that will at first just print what to do. Then you can use computer magic to improve things over time and automate steps in the middle. I generally agree that there are a lot of processes that are like "use your brain and think at this step", which is hard to automate, really. Open-ended workflow tools don't really exist for this kind of thing. (Clarifying example: what script do you write to "diagnose a response time spike"? You can definitely write up usual suspects but at one point you don't have a script that gets you from the beginning to end ) [0]: https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/ https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
  - mooreds 7y ago
    That's a great article. The HN discussion on it is worth reading too: https://news.ycombinator.com/item?id=20495739 https://news.ycombinator.com/item?id=20495739
  - epicgiga 7y ago
    This is a great intermediate step.
- scaryclam 7y ago
  The "Automate everything" mentality tends to come from developers who are not yet really experienced enough to know what value automation is bringing. I'm a developer and I love automation of things, but I also know that some things don't automate well (self-healing systems are great, but at this point in time they mostly fail in interesting ways, which is one of the reasons SREs exist). Automation also doesn't mean a system is actually of value. I've seen plenty of automation that has made life harder and incidents more common. It's also meant that some systems have been designed in a sub-par way, just to shoehorn automation in. The mantra "automate everything" should be a nice rule of thumb, but not followed with any dogma.
btmiller 7y ago
Automate everything and verbose runbook are not mutually exclusive. As you automate, your alerting events trend toward more complex issues that require understanding complex systems. In those cases, you don’t always know how much context the on-call engineer will have when receiving a page. That’s why I preach for verbosity and context setting in runbooks. They are extremely important for fast remediation of production-impacting issues.
mherdeg 7y ago
One of the advantages of having documentation live alongside code in source control is that, in principle, the person reviewing the pull request could edit the prose. The pattern I see most often is "oh my goodness, thank you for making any effort at all to document this mess, this is way better, your pull request is approved."
ggambetta 7y ago
> Runbooks are more of an anti-pattern than anything. No. They're essential to the sanity of the individual SRE oncall, and of the SRE team they are part of. It's also how institutional knowledge is preserved after successive rounds of oncall shifts. If you think a playbook is bad, try being oncall for the first time for a massively (Google-scale) distributed system without a playbook. > A list of commands you should run to accomplish X -> that's a script. A list of possible root causes an SRE should consider, complex interactions with other systems that might be problematic, not overreacting to spurious alerts -> that's a playbook.
- notyourday 7y ago
  > If you think a playbook is bad, try being oncall for the first time for a massively (Google-scale) distributed system without a playbook. You are not Google scale. Don't invent a pen that writes in space when a pencil would do the trick.
  - marcinzm 7y ago
    I'd argue 100% automation is a space pen while run books are a pencil.
    - notyourday 7y ago
      You do not need 100% automation. What you need is a systematic approach to handling problems followed by fixing the root cause. Runbooks came from techops in broadcasting, power plant operations, etc where there was a clear division between operators who pushed buttons, ran cables, etc and those that made decisions about buttons to push and cables to run. Dumb hands + runbooks created "smart hands". If your SRE runs like that it is not SRE. Look at the incident handling: 1. Identify the issue 2. Implement a workaround to restore the service 3. Identify the root cause 4. Implement a fix for the root cause 5. Remove the workaround Runbooks cover 1. and 2.
  - easymodex 7y ago
    Did you just assume my scale?
andrewflnr 7y ago
You still need documentation on what scripts to run under what circumstances and how, for instance flags and configuration. That's what your "runbooks" or whatever should be. I'm currently in the process of replacing as many procedural docs with scripts as possible, but the docs aren't going to disappear, even if they get reduced to mostly a shell command or two with some context. You're also making several assumptions like "everyone involved are devs" and "surely you don't have to manually plug and unplug wires, right?" Yes, my job is weird...
marcinzm 7y ago
>If accomplishing a task takes more than a single step, it means there's insufficient automation or programmability, and that's the problem you should fix, not teaching devs how to be better writers of prose. Perfect is the enemy of good. In practice your automation will not be good enough due to either tooling issues or script issues or time to manage automation for rare use cases. Worse, automation can also become stale and if it's for rare events then it's likely that it won't work when needed. After all, we all know that documentation is always out of date, why do you think the script won't be be? Run books solve that by involving a human and explaining to a human how to think about the problem and allow them to work around misinformation.
deleted 7y ago
[deleted]
peterwwillis 7y ago
You can't really automate everything. Even if you could, and you had the time, and the money, and the omnipotence to know of every possible potential failure and Domino condition and write automation for it before it happened, it would still be a bad idea. Computers can't replace the dynamic capability of humans to solve difficult problems quickly. But they can provide tools for humans to use to solve it quicker. Runbooks are just one of many tools humans use to this end, like scripts, like some automation, autonomation, escalation, incident management, etc. There will always be bugs no automation can fix, and so we rely on humans to fix it, and make it as easy for them to do that as possible. Furthermore, code is not documentation, because code does not explain how or why the decisions were made to write the code that particular way. Minimalism is not great when what you may need is knowledge. Documentation is different from a runbook. Documentation doesn't have to be tested, it just needs indicators of age and health. If it's >1yr old, it probably needs rewriting. A view count helps, as does a like button, this was helpful/unhelpful button, and comments section. If you link them, ones that change can indicate the others that may need changing.

mooreds 7y ago

> Automate everything. Whenever I see someone say something like this, I'm always curious about where human judgement and/or investigation comes in. Don't get me wrong, I'm happy to automate what can be automated if it makes sense (that is, the action can be easily turned into a script and the cost to create and maintain the automation is not higher than the value created) but aren't there plenty of areas where some level of human judgement is needed and you can't just automate everything? Isn't that kinda what SREs do? Develop intuition for what can be automated, automate what they can, set up process for what they can't?

epicgiga 7y ago
In the context of docs/"runbooks", my point is you're not saving anything by doing that instead of scripting. You're going to pay for it in the long run when the docs quickly drift out of date and then people stop trusting them, and the docs have limited value anyway because they're just going to ask for your help in person instead 90% of the time (because that's 10x more effective). Scripts can have similar problems, but they're rapid and testable, so far easier to maintain their integrity. Where judgment is needed, i.e. unique cases, the runbooks won't help much anyway, and its up to the tech lead to work out why that problem happened once they've fixed it, and then either modify the system, or add a new script to fix that particular new problem if it arises again.
rtpg 7y ago
I'm basically of the same opinion as you, but recently read this article[0] on "do nothing" scripts and really like it. The idea is to write at your process _as a script_ that will at first just print what to do. Then you can use computer magic to improve things over time and automate steps in the middle. I generally agree that there are a lot of processes that are like "use your brain and think at this step", which is hard to automate, really. Open-ended workflow tools don't really exist for this kind of thing. (Clarifying example: what script do you write to "diagnose a response time spike"? You can definitely write up usual suspects but at one point you don't have a script that gets you from the beginning to end ) [0]: https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/ https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
- mooreds 7y ago
  That's a great article. The HN discussion on it is worth reading too: https://news.ycombinator.com/item?id=20495739 https://news.ycombinator.com/item?id=20495739
- epicgiga 7y ago
  This is a great intermediate step.
scaryclam 7y ago
The "Automate everything" mentality tends to come from developers who are not yet really experienced enough to know what value automation is bringing. I'm a developer and I love automation of things, but I also know that some things don't automate well (self-healing systems are great, but at this point in time they mostly fail in interesting ways, which is one of the reasons SREs exist). Automation also doesn't mean a system is actually of value. I've seen plenty of automation that has made life harder and incidents more common. It's also meant that some systems have been designed in a sub-par way, just to shoehorn automation in. The mantra "automate everything" should be a nice rule of thumb, but not followed with any dogma.

btmiller 7y ago

Automate everything and verbose runbook are not mutually exclusive. As you automate, your alerting events trend toward more complex issues that require understanding complex systems. In those cases, you don’t always know how much context the on-call engineer will have when receiving a page. That’s why I preach for verbosity and context setting in runbooks. They are extremely important for fast remediation of production-impacting issues.

mherdeg 7y ago

One of the advantages of having documentation live alongside code in source control is that, in principle, the person reviewing the pull request could edit the prose. The pattern I see most often is "oh my goodness, thank you for making any effort at all to document this mess, this is way better, your pull request is approved."

ggambetta 7y ago

> Runbooks are more of an anti-pattern than anything. No. They're essential to the sanity of the individual SRE oncall, and of the SRE team they are part of. It's also how institutional knowledge is preserved after successive rounds of oncall shifts. If you think a playbook is bad, try being oncall for the first time for a massively (Google-scale) distributed system without a playbook. > A list of commands you should run to accomplish X -> that's a script. A list of possible root causes an SRE should consider, complex interactions with other systems that might be problematic, not overreacting to spurious alerts -> that's a playbook.

notyourday 7y ago
> If you think a playbook is bad, try being oncall for the first time for a massively (Google-scale) distributed system without a playbook. You are not Google scale. Don't invent a pen that writes in space when a pencil would do the trick.
- marcinzm 7y ago
  I'd argue 100% automation is a space pen while run books are a pencil.
  - notyourday 7y ago
    You do not need 100% automation. What you need is a systematic approach to handling problems followed by fixing the root cause. Runbooks came from techops in broadcasting, power plant operations, etc where there was a clear division between operators who pushed buttons, ran cables, etc and those that made decisions about buttons to push and cables to run. Dumb hands + runbooks created "smart hands". If your SRE runs like that it is not SRE. Look at the incident handling: 1. Identify the issue 2. Implement a workaround to restore the service 3. Identify the root cause 4. Implement a fix for the root cause 5. Remove the workaround Runbooks cover 1. and 2.
- easymodex 7y ago
  Did you just assume my scale?

andrewflnr 7y ago

You still need documentation on what scripts to run under what circumstances and how, for instance flags and configuration. That's what your "runbooks" or whatever should be. I'm currently in the process of replacing as many procedural docs with scripts as possible, but the docs aren't going to disappear, even if they get reduced to mostly a shell command or two with some context. You're also making several assumptions like "everyone involved are devs" and "surely you don't have to manually plug and unplug wires, right?" Yes, my job is weird...

marcinzm 7y ago

>If accomplishing a task takes more than a single step, it means there's insufficient automation or programmability, and that's the problem you should fix, not teaching devs how to be better writers of prose. Perfect is the enemy of good. In practice your automation will not be good enough due to either tooling issues or script issues or time to manage automation for rare use cases. Worse, automation can also become stale and if it's for rare events then it's likely that it won't work when needed. After all, we all know that documentation is always out of date, why do you think the script won't be be? Run books solve that by involving a human and explaining to a human how to think about the problem and allow them to work around misinformation.

deleted 7y ago

[deleted]

peterwwillis 7y ago

You can't really automate everything. Even if you could, and you had the time, and the money, and the omnipotence to know of every possible potential failure and Domino condition and write automation for it before it happened, it would still be a bad idea. Computers can't replace the dynamic capability of humans to solve difficult problems quickly. But they can provide tools for humans to use to solve it quicker. Runbooks are just one of many tools humans use to this end, like scripts, like some automation, autonomation, escalation, incident management, etc. There will always be bugs no automation can fix, and so we rely on humans to fix it, and make it as easy for them to do that as possible. Furthermore, code is not documentation, because code does not explain how or why the decisions were made to write the code that particular way. Minimalism is not great when what you may need is knowledge. Documentation is different from a runbook. Documentation doesn't have to be tested, it just needs indicators of age and health. If it's >1yr old, it probably needs rewriting. A view count helps, as does a like button, this was helpful/unhelpful button, and comments section. If you link them, ones that change can indicate the others that may need changing.