4 ms·
I’m not sure I agree with this viewpoint. Assuming nothing critical lived on the service, and the worst side effect of the deploy was the client failing once an
by Aeolun 7y ago
I’m not sure I agree with this viewpoint. Assuming nothing critical lived on the service, and the worst side effect of the deploy was the client failing once and reconnecting, I’m not sure there’s anything wrong with just restarting the world.
Obviously if you know in advance you should not even start, but if you notice after you are halfway done? Meh, chances are that a rollback would take as long as finishing the push, and either way the service disruption continues until it’s finished.
- clintonb 7y agoYou assume that is the only problem in the deploy. That assumption seems to have been correct. However, what if another bug had been found that requires a rollback from B to A? Now your rollback will result in a bunch of dead services until they are all on version A. The best option when a defect in B was detected would have been to rollback so that you aren’t forced into a scenario where every fix has to be a fix forward.
- michaelt 7y agoWhen I hear "It's one of those things that talks to other instances of itself" I think of a redundant, high-availability database. Assuming every machine sends data to all its peers regularly, if I have a cluster of 10 A machines and upgrade 1 machine to B, and the A machines crash, that means I lose 90% of my capacity - likely making the sole survivor get overwhelmed too. "Outages are OK, clients can always reconnect or try again later" isn't the usual standard HA databases are operated to :D
- Aeolun 7y agoAt that point the A machines are dead though. There is no point in restarting them as A when you can also restart them as B, so I assume they don’t all die in one go. But I guess we just have too little information to make a really informed decision about what you should do.
- klodolph 7y ago> There is no point in restarting them as A when you can also restart them as B, so I assume they don’t all die in one go. A is “baked in”, so it should be the default. There are simply too many classes of bugs which take time to manifest. Memory leaks, cache staleness, etc. So, for very good reasons, you consider version A preferable to version B.
- narag 7y agoWhen I hear "It's one of those things that talks to other instances of itself" I think of a redundant, high-availability database. Oh, I thought of an ancient copy of Windows for Workgroups taking down the Windows NT 4 domain servers every single day. Real story. But that case was version A crashing version B, not the other way around.
- kenhwang 7y agoMy rule of thumb once you're already in a bad state is to figure out the fastest way to get to a good enough stable state. Deploys/roll-forwards are very commonly worn paths, so it's usually pretty easy to speed up a deploy or just do nothing and let it run its course. Rollbacks tend to be special, unique, and rare which makes it difficult, slow, and high risk. Odds are you'll be able to finish the deploy and stop the damage way faster than you can rollback _and_ undo the damage. Obviously, avoiding deploying broken stuff is the best case, but preaching about fire prevention in the middle of a firefight is quite a bit too late and not terribly helpful.
- reaperducer 7y agoAssuming nothing critical lived on the service Why would you make that assumption? If someone is paying for a service, you are obligated to provide that service. It may be nothing more than another (for example) random point-of-sale system to you, but to the business on the other end, it could be fatal.
- taeric 7y agoI think the point is that is often not something you have to do on a blind assumption. Communication can clarify how critical a link is.
- dredmorbius 7y agoCorrupting persistent data or creating ambiguous and irreconcilable (or expensively-reconcilable) states is a Very Bad Thing and something which one should avoid with extreme prejudice.