4 ms·
Much of this is so overstated as to be wrong. > runbooks, etc actually provide little value in preventing incidents. There's a great deal of evidence that che
by mjb 7y ago
Much of this is so overstated as to be wrong.
> runbooks, etc actually provide little value in preventing incidents.
There's a great deal of evidence that checklists have value in preventing accidents of omission. In some systems we can build around needing checklists by adding automation, but in others we can't (or it's not worth it). Runbooks are good, but you shouldn't have too many of them.
> Unfortunately it turns out catastrophic failures in particular tend to be a unique confluence of contributing factors and circumstances, so protecting yourself from prior outages, while it shouldn’t hurt, also doesn’t help very much.
Things have causes. In complex systems there can be many causes, and the causes themselves can be systemic and complex. That doesn't mean that it "doesn't help" to defend against some of those causes. You shouldn't try to paper over every single little gap, but at the same time it's worth fixing some things. Adding mechanisms like runbooks, for example. Or backpressure. Or timeouts. Or unit tests. Or whatever.
There not being a single root cause shouldn't be taken to mean that you can't fix anything.
> The end result is that the additional bureaucracy inhibits the adaptive capacity of the organization and ultimately makes the system less safe.
Again, overstated. Of course too many procedures and too much bureaucracy is bad. But no procedures isn't the solution - depending on people's good intentions to do things is extremely failure prone, even if people have the best intentions in mind. Procedures are mechanisms that can help us avoid those failures.
> They prevent people from interacting with the risks in a system in a way that teaches them where the safety boundary actually is.
Just letting people break stuff is a great way to help them learn, but can also be an extremely expensive one.
> So rather than trying to simplify or remove complexity, learn to live with it.
I don't even know what to say about this. Sure, complex systems are complex. But accidental complexity is bad and adds risk.
> The number one myth we hear out in the field is that if a system is unreliable, we can fix that with redundancy.
This one seems right. Redundancy adds complexity (and hence risk), and only fixes unreliability if the cause of the unreliability is uncorrelated across the redundant systems. Is the cause of your unreliability uncorrelated? If it's random system failure, maybe. If it's software, no.