6 ms·
Keep calm and use the runbook
- zheng_qm 4y agoThis reminds me of the book The Checklist Manifesto and the show Air Crash Investigation.
- DarkWiiPlayer 4y agoMy first thought when I read the title was "Sounds a lot like the non-normal checklists they use in aviation" :D
- rkangel 4y agoI was also about to come here and say "this sounds a lot like a checklist". The Checklist Manifesto is one of those books that should be really boring but is both valuable and well written and I recommend to everyone.
- Tomte 4y agoThe book is overlong for what it has to say, and stuffed with things that are not really checklists (architect's building plans). Fortunately there is a better version with not just all the meat, but also the best anecdotes included: Gawande's original article in The New Yorker. https://www.newyorker.com/magazine/2007/12/10/the-checklist https://www.newyorker.com/magazine/2007/12/10/the-checklist
- citizenpaul 4y agoRunbooks the key ingredient to making any job unbearable. Sorta /s
- bartvk 4y agoSo what's your experience with them?
- citizenpaul 4y agoGeneric corp stuff TBH i'm not sure I"ve ever worked on an official "ITIL Runbook" but its all the same stuff with different names. I think they are a good idea in theory but they get taken to religion places with middle management and project managers types that want to make it part of their excuse for having a job. Despite that they are still never updated properly or accurately. Because when someone is tapping you on the should saying "where is the runbook update" while you are buried in more work than you can already do this week people just put something in there. THe people "enforcing" dont know anything or care to so they just update their checkbook/playbook as done. They also tend to be a "people are replaceable cogs" champion for the sociopaths in the company.
- bartvk 4y agoInteresting experience, thanks.
- hkt 4y agoNo /s from me. If you want a chore that will never pay off, write a runbook.
- strken 4y agoCan you explain more about this? Most of the runbooks at my current job exist because it was too painful to leave them unwritten, and get used frequently.
- hkt 4y agoRunbooks are, in my experience at least, generally sticking plasters over systems that are convoluted, unreliable, or otherwise broken. They're a symptom of teams that are running so hot that they can only put out fires, not prevent them. Again, just my experience, but whenever I see an extensive list of runbooks I run a mile.
- 300bps 4y agogenerally sticking plasters over systems that are convoluted, unreliable, or otherwise broken So pretty much every system not written from scratch in the past five years?
- hkt 4y agoNo, many older systems work perfectly well and are sensible and intuitive. Vertical scaling a single box or HA pair, for instance, is generally easier than working with distributed systems, for instance, and is perfect for many (not all) use cases. The idea that recent stuff isn't terrible is.. well, your anecdote against mine, I guess.
- politelemon 4y agoIt is obvious to me that keeping the runbook up to date is really important. But how do you foster a culture in which everyone keeps it up to date? Many people don't even read the readme, let alone correct it. Is it cultural? If you are so privileged that your books are updated, please share with us your ways!
- whateveracct 4y agoRound robin on-call
- lbriner 4y agoThis is the best suggestion imho. Most developers simply don't enjoy documentation because it doesn't "do anything" and good luck to those who say "employ people who will do it" because we all want to do that but when people don't do it, and you fire them, you have an even bigger issue with morale and rehiring. Putting people on an on-call rota gives them lots more experiences and takes them out of their silos and allows them to see the effect that their work has on everybody else. It also gives them the time to be updating docs without feeling guilty about not coding.
- toomuchtodo 4y agoWhat do you do in an org where developers aren’t included in the on call rotation and SREs are stuck with it alone?
- nijave 4y agoPost mortems/RCAs with after action items (that get followed)
- rozenmd 4y agoIt's cultural. Many companies will block folks from releasing "incomplete" work, so you end up with either no runbooks, or "complete" runbooks that go out of date incredibly quickly. Optimising for editing (https://onlineornot.com/incident-management/incident-response/guidelines-for-writing-better-runbooks#optimize-for-editing https://onlineornot.com/incident-management/incident-respons...) helps.
- gordonb 4y agoIt reminded me of a simple but not so obvious concept - do nothing scripts. https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/ https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
- nicbou 4y agoThis resembles autonomation in the Toyota Production System: https://en.wikipedia.org/wiki/Autonomation https://en.wikipedia.org/wiki/Autonomation Basically, you define a process, do it manually, then slowly automate bits of it.
- _qua 4y agoThis is a pretty neat idea. I'm an amateur programmer in addition to my job in medicine and I'm working in a new administrative role. There is so much being done by hand that I can see would be easily automated but, just as the blog post describes, I've been paralyzed by the all-or-none attitude towards doing so. This is a great way for me to start chipping away at some of these tasks.
- jamesdhutton 4y agoWhat is the difference (if any) between a runbook and a process? Genuine question; I am interested to hear people's answers.
- zwischenzug 4y agoRunbooks give more scope to human judgement and experience. But I don't know a formal difference.
- chiph 4y agoYou can have a process without a runbook, but not the reverse. i.e. a process can work, but be undocumented. Whereas having a runbook means you've defined the process (or at least gotten a good start at it..)
- tenebrisalietum 4y agoIMHO: - Process: follow it in order when you want result X, all steps need to be completed, each time every time. Routine stuff. - Runbook: follow it in order when X happens, some steps may not apply depending on the situation or you may need to adapt. Non-routine stuff.
- jillesvangurp 4y agoRunbooks are very common in safety critical environments. E.g. aviation is completely driven by checklists. Everything has a checklist. Even amateur pilots tend to stick to their checklists. There are a few useful properties with checklists: - They stop you from having to improvise actions when time is too short for that. People under stress take bad decisions. It's better for them to stick to a pre-defined plan. - They stop you from making preventable mistakes. People forget stuff. Even trained people. Especially routine stuff. - They ensure some level of uniformity to people's actions. So, they can check each other's work and do some cross-checking. For IT operations, run books are great. Especially if you can automate them. CI/CD is basically an automated run book. It simplifies the decision process for humans: do I want to put this live: yes/no? But what do you do when the database goes down or there is some major hardware failure and you have to restore from a backup? In a lot of organizations, this is still a major crisis. There might be a run book but chances are nobody has practiced this in production in ages. Thinking through ahead of time what you would do is more than half of the success. It doesn't matter if it involves some manual steps as long as you know what they are. In aviation, a lot of stuff is not automated at all. And they still use run books and checklists. These are a great substitute for automation. Newer planes self-check a lot of stuff that you used to have to do manually. Ideally you have both automation and run books. And automation can fail of course. What do you do when your cloudformation stack is corrupted or when your terraform is out of sync with what remains of your production environment? It happens and it's a bad time for experimentation. Usually such events lead to runbook updates. You learn some new stuff about exciting failure modes you did not consider before. Having the practice of documenting non obvious things is what separates a good from an excellent engineer. I actually do it to help my future self as well. Because I know I'll go down a rabbit hole of reinventing things I already figured out a few months down the line.
- slackerlol 4y agoThe “preventable mistakes” point reminds me of this great video about “normalization of deviance” [from routine/checklists] https://youtu.be/Ljzj9Msli5o https://youtu.be/Ljzj9Msli5o
- ofrzeta 4y agoIt's not nice but a perfectly usable runbook includes all required passwords :-)
- pseingatl 4y agoA European car company--that I can't name--proposes to take the owner's manual out of the glovebox, and thanks to the Internet of things--put it online. Their cars are exported to countries where there are Internet dead zones. What happens if you are in a dead zone and you need to look at the owner's manual in an emergency? Too bad.
- nexuist 4y agoWho needs to look at their owner's manual in an emergency? Nearly 99% of car manuals collect dust in the glovebox. Now that I think about it, I have never not even once seen someone pull a manual out of their glovebox. The only person that I know that has even read through the first few pages is myself, because I used to read my parents' manuals when I got bored as a kid.
- tootie 4y agoI think this is an outdated approach. If you have the wherewithal to write down precise procedures to operate your system, then you should just be automating them.
- mypalmike 4y agoIn the ideal world, yes. Some systems are, for any number of reasons, strongly resistant to automation. The obvious case is where security policies may require a human in the loop. Then there are the Rube Goldberg systems where the interface to the system involves VNC to a MacOS machine that runs a Virtualbox with a Linux guest where a native application modifies some database you have no other access to. Etc.
- kurupt213 4y agoStop publishing our trade secrets
- gbear0 4y agoI think my new favourite way of managing runbooks is to actually build them into a file tree of a bunch of simple python subcommand scripts, and have a run.sh script that scans the file system and uses argparse to construct a cli to call each script. # call ./runbooks/stack/update_secret.py # could update a secret in a vault, or update it in your deployed app ./run.sh stack update_secret --env=dev --name=foo --file=secret.txt Most of the time my python scripts are glorified CLI commands like `docker service update` that are called through subprocess, so you shouldn't need to install dependencies beyond what you'd be typing in the CLI. It's also easy to add a verbose option to print out the commands it runs so you can do it manually. # call ./runbooks/services/build.py ./run.sh services build -v > #--- Building images --- > #> DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --label "myapp" -t example-admin-ui:local "./admin-ui" > #> DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --label "myapp" -t example-frontend:local "./front-end" > #> DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --label "myapp" -t example-nginx:local "./nginx" Anything that can't be automated prints out an input line that gives instructions on what to do and just waits for you to input "yes/no" # call ./runbooks/get_crash_report.py ./run.sh get_crash_report --out=./crashes/ > # Copying crashes from AWS to './crashes/ > # Manual Step: Fill out crashes spreadsheet: docs.google/example_sheet > Continue [y/n]? The other really nice thing with this setup is the run.sh script is able to build up --help commands that can print out what actions are available and what params they use cause it's just python argparse. Makes discovery of what to do or looking up params really quick. At this point, the only culture you need to build is one where everyone's supposed to use the run.sh scripts and not do things manually. This enforces people to fix the scripts when something changes. YMMV, but I've found this has simplified a lot of processes for myself at least.