3 ms·

The best runbook is the one that doesn’t exist. Automating operational work vastly decreases risk, and gives you mechanisms to address many of the points in th

by cle 7y ago

The best runbook is the one that doesn’t exist. Automating operational work vastly decreases risk, and gives you mechanisms to address many of the points in this article (you can write audit logs, tests to ensure your automation works when you need it, etc.). If you need them, make them as short as possible. It’s an architectural smell if your runbooks are long and complicated—part of good design is making sure you can easily operate your system.

alttab 7y ago
What if the automation breaks? Zero ops is a good goal as.it builds the right incentives but even the most mature systems need a good run book.
- cle 7y ago
  That’s effectively the same as having an incorrect/out-of-date runbook. Except the likelihood of that happening with automation is much lower if you write tests to ensure they work when deploying. There are always some break-the-glass mechanisms that should be documented (like, how to log on to a host). I’m just saying that in mission critical systems, you can dramatically reduce risk by minimizing the amount of manual work you have to do, which prevents mistakes.
  - theshadowknows 7y ago
    I agree. Especially when you’re using vendor platforms for things. Salesforce is a great example. Sometimes things just stop working. And it’s a good idea to know which levers to pull and which buttons to press (and in what order) to get things moving again.
hinkley 7y ago
One of the better qualities in software developers is the ability to recognize when you’re being an apologist. And on top of that, learning not to fall back on that for everything. One of my exercises that I ask others to do too is first, ask yourself if it’s really easier to explain the workaround than fix the problem. Second, ask yourself if it’s collectively easier for everyone to learn the workaround or fix the problem. We have a really long run book for a process for a tool we’re breaking into pieces. Eventually we’ll have to run 90% of it three times, so it’s finally time to start automating the error prone parts.

by cle 7y ago

alttab 7y ago

What if the automation breaks? Zero ops is a good goal as.it builds the right incentives but even the most mature systems need a good run book.

cle 7y ago
That’s effectively the same as having an incorrect/out-of-date runbook. Except the likelihood of that happening with automation is much lower if you write tests to ensure they work when deploying. There are always some break-the-glass mechanisms that should be documented (like, how to log on to a host). I’m just saying that in mission critical systems, you can dramatically reduce risk by minimizing the amount of manual work you have to do, which prevents mistakes.
- theshadowknows 7y ago
  I agree. Especially when you’re using vendor platforms for things. Salesforce is a great example. Sometimes things just stop working. And it’s a good idea to know which levers to pull and which buttons to press (and in what order) to get things moving again.

hinkley 7y ago

One of the better qualities in software developers is the ability to recognize when you’re being an apologist. And on top of that, learning not to fall back on that for everything. One of my exercises that I ask others to do too is first, ask yourself if it’s really easier to explain the workaround than fix the problem. Second, ask yourself if it’s collectively easier for everyone to learn the workaround or fix the problem. We have a really long run book for a process for a tool we’re breaking into pieces. Eventually we’ll have to run 90% of it three times, so it’s finally time to start automating the error prone parts.