7 ms·
Night of the Runbooks: a DevOps horror story
- slowhand09 3y agoThey is responsible for this! And We is gonna make sure They pay for the time We lost working on It, to fix They's problem. We is out! No diggity.
- 8organicbits 3y ago> I don't know anything about process-payments; it belongs to a completely different team. I wouldn't even know how to check it. I get that the point of the article is to sell the value of run books, which are amazing, but are people really going on-call for the first time without any training or knowledge of the services they'll be supporting? Even with perfect run books that seems unwise.
- femiagbabiaka 3y agoUnwise but common. It’s a good sign of an org to stay away from IMO, especially if it means that the people creating services aren’t the people oncall for those services.
- jrace 3y agoI am 1.5years in to a new role, on call for 1.4 years so far... I still get calls about some service or application I have never heard of. Technical debt is real and I lose a little bit of my life every on-call week.
- yodsanklai 3y agoExactly my experience. I remember when I joined my company, after a couple of months I thought my job was pretty demanding but oncall added another level of stress. That being said, it's certainly less stressful now. At least, I can solve a bunch of issues myself, and I'm not lost in the internal tools anymore. And it's pretty rare to be bother in the night. And it's not like we're doctors or something. Worst case, somebody won't access a service for a few hours...
- somehnguy 3y agoI don't think it's that uncommon. When I joined my current org we spent much of my second week going through the apps we are responsible for, etc. But a week wasn't nearly enough to learn everything I needed to know about them. I was soon after added to the on-call rotation and received my first incident shortly after. A few weeks later, repeat. Each time I did what I knew and turned to my team to help with the rest - it has worked out fine so far and I learned so much more by getting hands on with failures compared to passively reading code & looking at diagrams. But I work in e-commerce, not life-critical systems of any sort..
- midasuni 3y agoMy org is large enough to have a dedicated 24/7 front line team that look after dozens (hundereda) of things. The churn in that team is high that you can’t really give them any significant training on every system. Even a 20 minute video would take weeks to onboard for a system they’ll likely never touch. As far as I’m concerned a run book is there to stop me from being needlessly escalated to at 3am. If there’s a genuine problem then sure, I or another SME need to be contacted. Most of the time there’s a problem it is a reduction in resilience (so no need for out of hours contact), or something out of our control (loss of power in a location, failure of a circuit, in which case the 24 hour team can raise with the appropriate people. I can count on one hand the number of problems worthy of a call out, and that’s in the last 7 years. As such the first entry in my run book is “judging importance”.
- tetha 3y agoI'd say this strongly depends on the kind of error and the quality of the application involved. Like, if the error is something technical - something is upset and refuses to talk with the database now, that's something you can identify and handle with minimal instructions. I'm working on a side-project at work to automatically generate dashboards for these technical foundations of services in a central discoverable place. Or, is the error something functional? Like, payments from Suaheli aren't being taxed right. There, a generic on-call has no chance. But why the hell would this not be caught in tests, or emerge at night out of office ours, and outside of a planned maintenance if you need that? And then you're getting to the quality and if your application is a humble workhorse or a feisty goat. With a lot of our systems, we've pushed it to the point so in doubt you can just restart an instance and it most likely will end up better. This makes on-call somewhat chill - in most cases, you just keep kicking the system one or two times overnight until more knowledgeable people come online in the morning and dump the problem in their lap. This includes out databases, just for the record. On the other hand, I've worked also with enough systems which need like 15 steps to put them back together after a restart."Oh, you need to stop this first, then restart that, then run this jenkins job, then start this first thing, then run this other jenkins job - it'll fail, but that's normal, ..." In such a place, some kind of generic on-call would be impossible. until you get these systems under control.
- hosh 3y agoThere was a company I interviewed at which had a particular way of hiring. Although I did not end up working for them, it was something I tried doing when I was involved in hiring for people with ops experience. And that is, being able to troubleshoot complex systems when you have no idea what is going on, under time pressure. Every site is a unique combination of side effects that no one on the internet — or even within the company — have any idea on. Yet there is generally a way to start approaching how to figure out what is going on. Though experience with the components of the system helps a lot.
- jachee 3y agoI call it Zero-knowledge Troubleshooting on my resume. It’s amusing how often, when I still worked in an office, that I would overhear part of a discussion, and some detail would tickle my spider sense, without even being sure what the problem being discussed was, and I’d ask a question or two, and it would help unblock the process. That sort of thing is literally the only thing I miss about the office.
- hosh 3y agoNeat term. I might steal that for my resume. Intuition does play a good part of it for me, though I have worked with good troubleshooters who are more methodical and rely less on intuition for these situations. But then again, having these different takes can help the team as a whole to figure things out and fix things.
- onionisafruit 3y agoI’m brushing up my resume today, and I would say there is a good chance this phrase will show up on mine too.
- macintux 3y agoRecent blog post along a similar vein. https://rachelbythebay.com/w/2023/05/30/eng/ https://rachelbythebay.com/w/2023/05/30/eng/
- 3y ago
- boruto 3y agoNot uncommon, We had a bank where they tried on call rota per group instead of team. It was horrible, A ping in at 2 AM meant I had to go through code to triage the issue. No amount of training is enough as features and code gets deployed every day. It was horrible and I left.
- throwawaaarrgh 3y ago> are people really going on-call for the first time without any training or knowledge of the services they'll be supporting? Yes. Very common > Even with perfect run books that seems unwise. Yes
- yodsanklai 3y agoCertainly very common in my company. After 1 year in my team, I'm still discovering services we're supporting that I've never heard of! But being oncall doesn't mean that you're supposed to fix every single issue by yourself. You're allowed to ask for help! Monitoring the alerts, taking care of some simple issues (broken CI), trying to fix what you can, and dispatching when you can't. This is usually doable by a new team member.
- sokoloff 3y agoI have taken on-call rotations with no training on the systems I'd be supporting. Even worse, I built and ran the human process that systematically put other devs into that exact same situation. We got good results from the process (as compared to not having the process), but it's obviously not as good as it could possibly be.
- sublinear 3y agoTo make this dream come true is far beyond the scope of just dev ops. It's also silly because if the problems are known this well the runbook will only be this good for a moment while the devs are working on fixing the bugs and the runbook will need to be updated constantly after each release. This requires more testing than most are willing to do and for dev ops to still always be on call to find the new bugs first and document them this way while waiting for devs to fix them and for management to always reserve time to keep this updated. This means there's a long enough release cycle for this to ever happen which is also unlikely.
- namaria 3y agoIn my experience, this is signaled by companies hiring "devops engineers". It shows lack of understanding that "devops" means devs and ops people should work together. And usually means they want someone to hold the bag because they can't seem to make their system work well, for some reason.
- GauntletWizard 3y agoI highly recommend against runbooks. There should be no need for runbooks. Anything that can be identified down to a single solution should have an automated response. What you should have is hintbooks. For each alert, have something that identifies why you made the alert. What dashboards to check, what logs to read, what else might be going wrong.
- ryanklee 3y agoAutomated responses take time and tech to implement. Playbooks are a way to iterate into automation. Good automated solutions often don't emerge fully baked.
- GauntletWizard 3y agoBy the time you have enough clarity to create a "playbook", you probably have enough clarity to reproduce and to automate. That said, there are still a ton of solutions where "playbook" is kind of the right call. "DB is out of Disk Space" is probably "Increase the size of the disk reservation" for most modern cloud-native companies. They rarely rise to the level of detail that proponents of "Playbooks" suggest, though. How do you do that? Is it clicking through the UI? Do you have the CLI command prepared? Have you thought about why disk space usage has grown suddenly? This is the sort of thing that may be a no brainer you were expecting (in which case - Why didn't you complete the operation in-hours?) or it could be the "That's funny[1]" moment that precedes a major discovery. [1] https://quoteinvestigator.com/2015/03/02/eureka-funny/ https://quoteinvestigator.com/2015/03/02/eureka-funny/
- bad_username 3y agoAs a non-native English speaker, I still find it difficult to read texts where a singular person is referred to by a plural pronoun. Every "they" becomes ambiguous and requires more mental cycles to unpack. Is it about Morgan, or some unrelated group of people, or Morgan plus the group? I understand the cultural reasons, but I regret the loss of linguistic clarity.
- p_l 3y agoI'd blame modern teaching material, because "they" is both singular and plural pronoun - just somewhat recently avoided in style guides.
- smeagull 3y agoloss? It's been English usage since Chaucer. The "rule" that it shouldn't be singular came from the same people that changed English spellings because they wanted to show off their knowledge of Latin and Greek. All those difficult words that are irregular and don't make sense? Those people.
- namaria 3y ago> As a non-native English speaker, > I understand the cultural reasons, but I regret the loss of linguistic clarity. Have you thought about the fact that you're using ignorance of English to complain about a very specific anglophone piece of politics?
- bestest 3y agoI just think this is written by an AI. It just feels off somehow, yet I can't pinpoint it exactly.
- kefabean 3y agoI agree, the ‘punchline’ is also an anti-climax and doesn’t seem to offer anything novel.
- Makhini 3y agoAnd rightly so, because noone writes prose like that. It sounds unnatural.
- rjmunro 3y agoI thought it was going to turn out to be a sophisticated phishing attack.
- midasuni 3y agoWhat did Morgan (in the dream) actually accomplish that a small shell script couldn’t? Literally each step was “if error then do automated action” Alas I have emails which go to a circuit provider when their circuit goes down as they can’t seem to monitor it. It does automated triage, for example checking my router is up and powered, checking the link light on their adva etc. It tells them there’s no power outage. They inevitably ask the first question. “Is there a power problem” A run book, or any instructions, is only as good as the person following them.
- throwawaaarrgh 3y agoAs expected, this is an Ops story, not DevOps
- amelius 3y agoLooks more like a meme-hater's horror story.
- larsrc 3y agoAnd here I was waiting for it to turn out to be a phishing attempt due to clicking the link in the alert.
- mynegation 3y agoI was baffled by the lack of actual troubleshooting. Ok, so graphs show increased failures, so what? What is happening? Spike of inbound traffic? Hardware failure? Increased connection pauses because of DNS? Database migration mishap? Without at least understanding the underlying reason on the high level restarts and version rollbacks may not help or may even aggravate the situation. Am I missing anything here?
- morelisp 3y agoThis is the most normal part, at 3AM you do whatever is necessary to get stuff working only until 9AM. Then at 9AM you have a real team to work on it.
- GRBLDeveloped 3y agoAnyone here have buttons in runbooks? We normally have links to Jenkins jobs in git runbooks, what are you using?
- tfolbrecht 3y agoThe one place I've set up button triggered runbooks was gitlab ci under scheduled jobs. Made things a lot easier for less technical people.
- ang_cire 3y agoWait, is the idea that your SRE/ Ops person doing rollbacks, failovers, and restarts without understanding why... is a good thing? Runbooks are supposed to give you a clear path during an incident so you don't run down rabbit holes, but you still shouldn't be running them without understanding what the impact of their steps actually is. I thought the horror part was that some random person, who doesn't know the system, was doing all this stuff without just contacting the people who do know it. If the runbook was sufficient to handle it, great! The proper team will have just as easy a time doing the steps, and not be mucking around in something they don't own and won't be the ones to fix when the runbook process itself breaks something because Error A is not Error J, but un-familiarized person can't tell the difference.
- theK 3y agoThe way I read it the horror was reading how that innocent guy was performing actions "they" didn't really understand without the setting itself being conclusive enough that "they" could actually trust those instructions blindly. Did I read it wrong?
- ang_cire 3y agoI think the ending is supposed to flip it into being that the person not having that runbook is the horror part.
- brainzap 3y agoit depends if the company designed it that way, in a bigger company I would have the policy that all deployments can be rolled back (except some bigger migrations)
- ang_cire 3y agoSure, but what if an infra change that is not part of that automated deployment also needs to happen? It's one thing to allow anyone to *call* for a rollback of a deployment. It's another thing entirely to allow anyone to, independently and without sign-offs, *perform* a rollback.
- wkdneidbwf 3y agohaha, it makes sense this is on a site with “consulting” in the name—its unrealistic and pointless. the M. Night Shyamalan twist is that the whole thing is the “horror” portion. in the story our protagonist has no knowledge of the service she is on call for and displays no ability to troubleshoot, reason about, or understand the issue she’s responding to. instead, she clicks a series of buttons in a runbook to resolve the most basic and most happy-path production issue you’ll ever see. aaaaand this is about devops how?