3 ms·

‘Prevents an issue like this: "I recently ran into a situation where I spent 6 hours understanding how something works that would have taken 20 minutes if the r

‘Prevents an issue like this: "I recently ran into a situation where I spent 6 hours understanding how something works that would have taken 20 minutes if the relevant information was stored somewhere."‘ Recently I spent 10 hours on a usually routine task. At the end of the slog, my first thought wasn’t “I should spend more time writing that up!” The article was a good reminder that it’s worth scribbling something down. Even the basics of the “gotcha!” and snippets of code for debugging could save someone else (future me?) another 10 hours. One thing I didn’t get from the article was how runbooks are created. It mentions the “sticky note on someone’s desk” approach and the “workflows for everything” approach. There’s a lot of ground in between. I guess people write lots of how-tos and eventually they’re turned into a runbook?

Normal_gaussian 5y ago
I don't write enough 'runbooks' (I call them playbooks). But I do write them, normally in four ways: - every three months I spend a day updating my 'what can go wrong' documents. List of issues, explanation of effects, possible technical and human solutions. A lot of these link to one or more playbooks, but some do not. - when a new system is created or a long process discussed I'll throw the links and thoughts in an untidy playbook or a nearby README, whichever feels more appropriate. I'll tidy it up more if its going to be part of training. - Randomly when I think about something. - during an incident, or it the post-incident handling. -- When actually writing the book I give an overall guide to whats going on in prose at the top, then several sections with relevant things. It might be a series of commands to run, a list of things to bear in mind, a series of links to relevant documentation/blogs/etc, or a mix of the lot. Again, as appropriate. All the playbooks are in a repo of markdown files whoch gitlab renders and supports crosslinking. I do use a greasemonkey script to give copy+hide hover buttons on all scripts, as well as making large scripts scrollable, which are great ease of use features I haven't seen any more concrete solution do.
cle 5y ago
For my teams it's always been a cultural thing. Whenever something comes up, someone (at first, probably you!) has to ask: "would a runbook have helped here?". If so, give the person time to write a quality runbook. Also when things change, you have to remember to update your runbooks. Also wanted to mention something which I always harp on when runbooks come up: the best runbook is no runbook. They're necessary for some break-the-glass things, but if the answer to "would a runbook have helped here?" is "yes", then the immediate follow-up question should be "should it be automated?". If the answer is "yes" then build it into your system, with tests and everything, and if that's not feasible then write a script for it and check it into VCS somewhere.
- deleted 5y ago
  [deleted]
wsinks 5y ago
I'm finding that the best way for me to document is to realize that I'm going to go slower at everything but I'm going to document along the way. I'm in Events, and typically in an event it's been low payoff to try and do anything _after_ the event. Each event is a burnout because you're always pushing yourself. I assume the same is in software too. I can't find the original article, but I often find that keeping a running note or checklist or even a pseudo-script (written like a script but not executable) are really helpful. I see what you're saying though, my run book right now is just a collection of articles that I've written and occasionally refactored to make more sense as a whole.

Normal_gaussian 5y ago

I don't write enough 'runbooks' (I call them playbooks). But I do write them, normally in four ways: - every three months I spend a day updating my 'what can go wrong' documents. List of issues, explanation of effects, possible technical and human solutions. A lot of these link to one or more playbooks, but some do not. - when a new system is created or a long process discussed I'll throw the links and thoughts in an untidy playbook or a nearby README, whichever feels more appropriate. I'll tidy it up more if its going to be part of training. - Randomly when I think about something. - during an incident, or it the post-incident handling. -- When actually writing the book I give an overall guide to whats going on in prose at the top, then several sections with relevant things. It might be a series of commands to run, a list of things to bear in mind, a series of links to relevant documentation/blogs/etc, or a mix of the lot. Again, as appropriate. All the playbooks are in a repo of markdown files whoch gitlab renders and supports crosslinking. I do use a greasemonkey script to give copy+hide hover buttons on all scripts, as well as making large scripts scrollable, which are great ease of use features I haven't seen any more concrete solution do.

cle 5y ago

For my teams it's always been a cultural thing. Whenever something comes up, someone (at first, probably you!) has to ask: "would a runbook have helped here?". If so, give the person time to write a quality runbook. Also when things change, you have to remember to update your runbooks. Also wanted to mention something which I always harp on when runbooks come up: the best runbook is no runbook. They're necessary for some break-the-glass things, but if the answer to "would a runbook have helped here?" is "yes", then the immediate follow-up question should be "should it be automated?". If the answer is "yes" then build it into your system, with tests and everything, and if that's not feasible then write a script for it and check it into VCS somewhere.

deleted 5y ago
[deleted]

wsinks 5y ago

I'm finding that the best way for me to document is to realize that I'm going to go slower at everything but I'm going to document along the way. I'm in Events, and typically in an event it's been low payoff to try and do anything _after_ the event. Each event is a burnout because you're always pushing yourself. I assume the same is in software too. I can't find the original article, but I often find that keeping a running note or checklist or even a pseudo-script (written like a script but not executable) are really helpful. I see what you're saying though, my run book right now is just a collection of articles that I've written and occasionally refactored to make more sense as a whole.