9 ms·
I'm going to have to respectfully disagree with a big chunk of this article. Documentation is generally a waste of time unless you have a very static infrastru
by jedberg 10y ago
I'm going to have to respectfully disagree with a big chunk of this article. Documentation is generally a waste of time unless you have a very static infrastructure, and run books are the devil.
You should never use a run book -- instead you should spend the time you were going to write a run book writing code to execute the steps automatically. This will reduce human error and make things faster and more repeatable. Even better is if the person who wrote the code also writes the automation to fix it so that it stays up to date with changes in the code.
At Netflix we tried to avoid spending a lot of time on documentation because by the time the document was done, it was out of date. Almost inevitably any time you needed the documentation, it no longer applied.
I wish the author had spent more time on talking about incident reviews. Those were our key to success. After every event, you review the event with everyone involved, including the developers, and then come up with an action plan that at a minimum prevents the same problem from happening again, but even better, prevents an entire class of problems from happening again. Then you have to follow through and make sure the changes are actually getting implemented.
I agree with the author on the point about culture. That was absolutely critical. You need a culture that isn't about placing blame but finding solutions. One where people feel comfortable, and even eager, to come out and say "It was my fault, here's the problem, and here's how I'm going to fix it!"
- zwischenzug 10y ago'Documentation is generally a waste of time unless you have a very static infrastructure' I definitely agree with that, and it's partly a corollary of 'documentation is expensive and requires costly maintenance'. Run books/checklists are mostly implemented really really badly. Automation is the ideal, but is costly, and itself requires maintenance. Most of the steps we had to perform did not lend themselves to automation, also.
- jedberg 10y ago> Automation is the ideal, but is costly, and itself requires maintenance. I would contend that the cost of automation is about the same as the cost of documentation plus the cost of having to manually do the work over an over. It's just a cost borne up front instead of over time. But to your point in the article, you have to have a culture that supports bearing that up front cost. > Most of the steps we had to perform did not lend themselves to automation, also. I don't understand how that is possible? Could you give an example of a task that can't be automated?
- zwischenzug 10y agoYour arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason. Also, there's no point automating something that happens once a year - the cost will exceed the benefit. Hence the comments about decided which tasks to automate with a backlog and metrics. Most anything _could_ be automated, eg 'Phone the customer and ask them to engage their network team', 'look for similar-looking strings', 'tcpdump the port, and see if the output is irregular', but is it really worth the effort to? That's where the backlog came in. BTW I wrote a whole framework for automation ianmiell.github.io/shutit/ and it's taught me a lot about the hidden costs of automation...
- jedberg 10y ago> Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason. If everything in your infrastructure is deployed with code, then automation is simply the act of making sure the infrastructure matches the deployment described in the code. It's true that automation is just as difficult if deployments are manual. Then remediation becomes changing the deployment code to fix a previously unknown problem, instead of manually fixing a problem. This then gains you the advantage of the problem being fixed in perpetuity. So they aren't in contradiction if your remediation and deployment are the same process, because then it by definition is always up to date. I guess I should add the caveat that deployment should be quick enough to solve problems via redeployment.
- geofft 10y agoMost of the environments I've worked at have been so far from being deployable with code that it would take you significant time and expense to get there, and in the meantime, you need to keep everything working. You need a reliability engineering team that knows how to triage problems and keep the existing system working while they automate what is feasible to automate. I'd probably agree that this is a separate mindset and skillset from keeping an infrastructure-as-code deployment running, and it's unfortunate that we don't have good terms to differentiate them. A lot of the SRE mindset applies, but not all of it. For instance, if you've got machines at customer sites where you manage the software, commercial filers or appliances where you don't really have a shell and you certainly can't spin up a VM, etc., your machines aren't fungible. You can't just redeploy a machine, and just about every problem on the machine is going to be a new one. You want the runbook so that it enumerates how to stop and start things around a human intervention (and yeah, as much of that as possible should be automated), but you can't automate the whole thing without changing your architecture. If you have the option of starting with an architecture where you don't have these problems, by all means take that option! Maybe do cloud-hosted SaaS instead of maintaining on-prem software for customers, or use some fancy cloud storage API instead of a physical old-school filer. But people who have the old-school architectures need to keep things running smoothly, too.
- closeparen 10y agoA skilled operator reading a runbook would seem less likely to exacerbate the mess than a naively written script. The operator can think, "wait a minute, this isn't right." Maybe they won't when they're sleeping off a night of drinking at 3am, but they could. The Python interpreter can't. To make the script less naive, you'd need a development environment with the same infrastructure components as production, and the freedom to make them fail so you can develop your script against their failure states.
- dastbe 10y agoThen have that environment? If you know what you need and it isn't unrealistic, then get/make it.
- gdulli 10y ago> I'm going to have to respectfully disagree with a big chunk of this article. Documentation is generally a waste of time unless you have a very static infrastructure, and run books are the devil. I estimate there's maybe 25% of the industry that's ignorant of "best practices", 70% that follow them dogmatically, and 5% that use them as guidelines, evaluating each situation on its own merits and choosing what makes sense. I feel like it's more frustrating to deal with the 70% than the 25%. For some people documentation can only sound like a virtuous thing. But documentation can do harm as soon as it gets out of sync with reality.
- wink 10y agoThere's stuff where you can either spend weeks or months automating it - or live with the fact that your oncall engineer has to use a runbook a few times per year to fix the problem. Also your Netflix example - how many people do you have there? Probably the smallest of teams is bigger than my company's whole engineering department. We're running a whole company and not a few services. (I'm absolutely not trying to discourage what you're doing - but I strongly feel it's a different ballgame.) The smaller your team with too many widespread responsibilities (not: make THIS service be available 99.9%, but make ALL services available 90%, then 95%, ...) the more you only automate what happens frequently. And yes, we try to first a basic runbook when something happens - and only if the same thing happens repeatedly, we automate it.
- jedberg 10y agoWe were doing this with a team of sometimes one or two reliability engineers, but we were cheating, because our company culture meant that the engineers who built the systems are responsible for keeping them running, so they would invest their engineering time in fixing the problems along with us. I personally found that runbooks were even worse for small size teams (like our four person reddit team) because they would get out of data even quicker than at the bigger places due to the rapidly changing environment. I wrote down thread that if all of your deployment is automated than it is much easier to automate remediation, because you just change your deployment to fix the problem, as long as you can redeploy quickly.
- wink 10y agoI agree on one level (literally, not figuratively) - if you have awesome (or even bearable :P) deployment for your software stack. We can do that. On the other level, there's stuff we see as core infrastructure (for example Hardware, or even some parts of OpenStack running on that hardware) - of course there are also downtimes and emergencies and dumpster fires - but they are pretty much unique little snowflakes and the repeating ones happen a few times per year. There simply is nothing "to deploy". Maybe one can argue that "runbook" is not 100% correct, sometimes it's a runbook including debug info. But it's not turtles all the way down, and I stand to my point there's stuff where the cost to benefit ratio totally ends up at "automate it away" and there's other stuff where it's the opposite.
- BuuQu9hu 10y agoDocumentation is the only defense against tribal knowledge. If your turnover is low, you might not notice this, but that doesn't mean that it isn't biting you. Not all processes can be automated today. Some can only be automated tomorrow. Some can only be automated after some blocking functionality is added. Documentation is how you plan your automation.
- mianos 10y agoIn the nineties.
- emorse 10y agoAs far as this discussion goes, I think that code is documentation. If you need to figure out how something works, going directly to the automation code will give you the right answer. If the code is difficult to understand, that's more of a quality issue, and the same problem happens with word documents.
- robomc 10y agoCode as documentation only works if all your infrastructure is managed programatically, there are good tests, the code is good, and overall complexity is low. I mean, if your downstream partner calls up to say that the custom analytics feed from your xyz service is returning null data, but not erroring, and the guy that implemented that feed (with rolled eyes, thinking it was an inelegant hacky concession to a noisy customer) left in 2015, where do you even start? How much code from how many codebases and configuration management repos are you going to have to read through just to work out what kind of problem you've got? Some type of high level documentation - what services / products exist, what infrastructure does each use, how is each one managed and tested, what is worrisome about it or what has tended to go wrong in the past - is going to help a lot.
- chimeracoder 10y ago> As far as this discussion goes, I think that code is documentation. "Code as documentation" can be an okay answer if the question is "what is the behavior of this system"? But it's a bad answer for the question "what is the intended behavior of this system, and what assumptions does it run under?" Looking back at a piece of code months (or years) later and not knowing if a particular edge case that you're running into was actually the intended function or not is not particularly fun. The power of writing documentation is not just in the end product; it's that it serves as a forcing function for the developers to confront their own thought processes and make them explicit. It's possible to write code that makes all of its assumptions explicit and clearly states its contracts up-front, but in practice, it almost never happens without significant external documentation (whether that comes in the form of explicit code docs, whitepapers, or ad-hoc email threads and Slack conversations that need to be pieced together after-the-fact).
- joncrocks 10y agoI have a question. The fact that an incident review occurs suggests that there was some kind of incident, and one that presumably wasn't anticipated and automatically handled. What does one do in that kind of situation where one has a novel problem and needs to do something about it? Should the organisation keeps all of the details of how to do things and how things work in their heads (and presumably ensuring that enough people are available whenever you might have an issue), or work out how things work/what does what on the fly? (or maybe a mix of the two?)
- jedberg 10y agoWell, ideally the person who wrote the code is fixing it, and should know how it works. Or at least a close teammate. But if you don't have that luxury, then it means you're hoping that the documentation is up to date, but chances are that if you don't have enough resources to have on call coverage from someone very familiar with the area that's having a problem, then there is a good chance they didn't have time to write or update documentation. So ideally the person who wrote it has it in their head, or you're figuring it out on the fly and hoping you have good comments in the code and good metrics.
- zwischenzug 10y ago> Well, ideally the person who wrote the code is fixing it, and should know how it works. Or at least a close teammate. What if the code is 15 years old, and everyone involved with writing it has left?
- vacri 10y ago> Documentation is generally a waste of time unless you have a very static infrastructure I find that while documentation suffers from 'detail rot' quite fast, it does help with higher-level stuff like the rough outline of how things are organised and where they live, or why decisions were made to do -foo-.
- Thaxll 10y agoDocumentation is really important especially for systems that will run for years, you will get new employee that have no ideas about how things work.
- CodeWriter23 10y agoIn my experience, trusting five year old documentation is a huge risk in and of itself. Having been in the position you put forth, you can use the documentation as a road map of the developer or architect's intent at a certain point in the past, but you have to start reading code in case it was changed and the documentation was not updated.
- zwischenzug 10y agoWhich is why maintaining it is a critical investment.
- CodeWriter23 10y agoHow do you know it is up to date? The only thing you can assume is it isn't.
- zwischenzug 10y agoIf the process is correct, and properly followed, then it must be up to date.
- Spooky23 10y agoI'd argue that your incident review/debrief process is where you are doing your documentation after the fact.
- jsmthrowaway 10y ago> You should never use a run book -- instead you should spend the time you were going to write a run book writing code to execute the steps automatically. While I hesitantly agree despite it seeming counterintuitive (you make a good case), I'd contend the code can take a form that looks runbook-y. I've had success in my organizations with Jupyter notebooks with documentation mixed throughout. Sometimes you do need a human, and in those cases having the documentation update live with the state of the world was huge for comprehension, particularly when you're centrally executing the notebook. Each step is something like: > 0) Blurb about what's going on, warnings, etc. > 1) Call into your automation code with one well-named entry point, like reboot_all_frontend_servers(). > 2) Display the relevant results immediately under that cell. Then you can step, yadda yadda. Idempotency of the steps is key. I have a vision for the operations bible taking such a form, with each thing-ops-needs-to-do represented with a notebook, but that might be unattainable -- you might be correct about that. Even still, mixing documentation and code seems to potentially push that barrier just a little further back. As a few people told you elsewhere in this thread (and, I think you know), complete automation is a big ask outside of a subset of maybe a dozen valley companies, even at a small property level. Until then, giving humans the tools to reliably do their job, like mechanized dynamic (not static) documentation, might be useful enough to not discard entirely.
- otterley 10y agoWhat about making updated documentation a required gate for the release process? Wouldn't that keep it from going stale?
- aanm1988 10y agoI'm working on a system with a stupid amount of poorly documented/undocumented moving parts. It's all tribal knowledge and blocking trying to hunt down the right person to bug.