4 ms·
Lot of answers so far seem to have missed the points being made in the book the article references. If you have the ability to ship things fast - and so do reg
by PaulRobinson 7y ago
Lot of answers so far seem to have missed the points being made in the book the article references.
If you have the ability to ship things fast - and so do regularly - you will find that your releases:
- Have a small blast radius if there are bugs, because how much blast can you have in an hour's work versus a quarter's?
- The tooling that you build to let you go fast will tell you that a blast is indeed happening right now
- Your Mean-Time-To-Resolution (MTTR) is lower because you can just roll it back. It's not days or weeks of work to get a rollback out, it's minutes.
- That having a culture of being able to release continuously and quickly allows you to go fast, and the occasional breakage is tolerable, because the cost is low.
You can go slow if you want. And if you don't have LEST metrics or have slow release processes you must, but if you work to get to the point where you are able to move fast, you'll find that you break things less and fix them quicker when they do break.
The alternative is you still break things, but the problems incurred are much, much greater.
Also note: there are some environments where continuous deployment is never going to be an option or desirable. Thankfully these environments tend to be ones where the market will tolerate the higher costs of formal methods and a more deliberative quality control process (medical devices, aerospace, nuclear reactors, etc.).
- altcognito 7y agoI like everything except the small blast radius point here. You can do a lot of damage in a short period of time with computers.
- PaulRobinson 7y ago`sudo rm -rf /` is a very big thing to do in a very small amount of code, sure. I'd adovocate thinking about probabilities then: If you have a shorter period of time to produce a smaller change, are you more or less likely to have a smaller blast radius than if you have a longer period of time to produce a much larger change? There are no absolutes, but we're talking about risk here, and risk is about impact and _likelihood_. Your impacts are going to be smaller and your likelihood of big blast radiuses are _less likely_, but not ruled out.
- zepto 7y agoI’d say that change duration is irrelevant to blast radius, because bug impact is independent of code size. Having said that - a smaller piece of code is easier to understand, so there is more potential to notice problems. However that depends on whether you do that diligence. And the dangerously false belief that the blast radius is going to be smaller because of change size rather than understanding works against you.
- LgWoodenBadger 7y agoIf you assume that any and every change could have a disastrous effect, would you rather have to deal with one change at a time or many changes at a time?
- zepto 7y agoI’d rather people not think that the risk is lowered because of the code size. It’s a red herring. Risk is lowered because the code is understood better not because the code is smaller. The assumption that each change carries an equal risk of disaster isn’t valid, because risk depends on your understanding of the changes and the codebase. Risk is equal only in the case where the developers know nothing about the change or the codebase. In this case, fire the VP of engineering or equivalent. Making changes as a series of small increments can just as easily lead to less understanding of the overall impact of the changes than a larger design process. Then you get a nice big failure from a small change. It looks like most of the changes were harmless when in fact they were all just building towards the one that caused the failure.
- roosterdawn 7y agoI don't know whether a smaller change means necessarily a lower probability of small blast radius, but the larger a code change is the higher the probability is of a high blast radius. Blast radius isn't necessarily going to be smaller because of change size, but the chance of it being smaller certainly does decrease in relative terms as it becomes smaller even if it never reaches zero and is still in absolute terms quite high. It's still a noticeable improvement and because of that still a worthy goal to pursue, nevermind the ancillary improvements to product/design/engineering/business coordination and delivery volume as a result of lower iteration cycle time. That's the larger win for me.
- toddh 7y agoPrograms can be very efficient disaster compression algorithms.
- jasoncabot 7y agoWhat are LEST metrics? I couldn't seem to find anything from Google and now it feels like I'm missing some major concept
- PaulRobinson 7y agoLatency, Error rate, Saturation (think: request queue depth), and Throughput. They are the “Golden Signals” in Site Reliability Engineering (SRE).