4 ms·
The author of the blog starts with the claim that you can't have a rollback button because applications aren't self contained and then at the bottom of the arti
by skookum 10y ago
The author of the blog starts with the claim that you can't have a rollback button because applications aren't self contained and then at the bottom of the article proceeds to outline parts of how rollback "buttons" in distributed systems are enabled. A rollback button is rarely a "button" and is instead a procedure that was reasoned through in advance of deployment.
Every CM at every large cloud vendor/distributed infrastructure shop has a field asking the operator to detail the rollback procedure which is usually a set of steps that include both turning off new code and fixing up state. Moreover, virtually all large scale distributed systems need to be able to run in mixed mode with old versions and new versions co-existing in different parts of the fleet for hours or days at a time. It's simply not possible to flip from one version to the next in any way that even loosely approximates atomicity at the fleet scale of Google, Amazon, Facebook, etc. Any competent distributed systems engineer will think through the upgrade implications... "How does Vnext RPC to/from Vcurrent?", "How will Vcurrent handle changes to central database state made by Vnext?", "How will Vcurrent handle local on-disk state if I need to flip back to it from Vnext on some set of nodes?", and so on. Often such upgrades entail multiple deployments where backward compatibility code is pushed ahead of RPC/state changing code.
Does all that mean there aren't occasional mistakes rendering a planned rollback impossible and the only path forward is to soldier through to V(original + 2)? Of course not. But the author's idea that you should generally plan to soldier through to newer code if things go wrong during deployment as a standard operating procedure betrays a lack of experience and sends a dangerous message. When deployments go off the rails is exactly the time when follow-on mistakes are made - at that point you've got a tense conference call going, the managers/PMs/VPs are asking for ETA for the restored system and the only acceptable reply is "very soon", the engineers are trying to diagnose quickly and may overlook some symptoms of the botched deployment, etc. Even the best engineer is not going to do a great job thinking through all the implications of the remediation they are attempting at that point and the code they're now about to push has almost certainly cut a bunch of corners in QA. That's not a situation I willingly put myself in.
- tarmstrong 10y ago> But the author's idea that you should generally plan to soldier through to newer code if things go wrong during deployment as a standard operating procedure betrays a lack of experience and sends a dangerous message. The author says "reverting smaller diffs as a roll-forward is more verifiable" near the end of the article. I agree the title makes this a bit confusing, but I don't think he's arguing that the only way to recover is to write a patch under pressure.
- skookum 10y agoReverting smaller diffs out of order to roll forward IS a patch under pressure. Usually most interesting features, the kind that tend to cause bugs that are not caught until deployment time, tend to come in as several patches over time. So reverting those means cherry-picking a set of patches, making sure you got the full set, making sure no subsequent patches took some subtle dependency on the ones being pulled, and so on. If you're in a compiled language, the set of patches is not too big, and the churn on your repo is low, then you probably have a decent chance of getting it right under pressure even if you have to untangle some refactors and such. If not, good luck.
- kelnos 10y agoReverting a smaller diff is writing a patch under pressure. Reverting a section of your new code is just as difficult as authoring a new fix. You need to be just as sure that the partial revert will not interact poorly with other parts of the new code that you are not reverting.
- tarmstrong 10y ago> You need to be just as sure that the partial revert will not interact poorly with other parts of the new code that you are not reverting. By partial revert, I'm imagining that three people have changesets (A, B, C, in that order) that have been deployed. You notice that A broke and you make A' to revert it. I think the author is arguing that it is easier to review A' to see if it is a safe change than it is to verify that A', B', and C' (the full revert) are safe to revert. In other words, even if you don't use version control to record that you reverted A, B, and C, you still effectively do that by reverting in full. You just know that the combination of A', B', and C' was safe when it was deployed. Is that what you're imagining or are we talking about different things? (I don't have strong opinions about this, I just want to make sure I understand your perspective (: )