3 ms·
Large-scale deploy systems for a distributed environment are ridiculously hard to build. We had to set one up quickly and on a budget, so in lieu of fancy engin
by paxys 7y ago
Large-scale deploy systems for a distributed environment are ridiculously hard to build. We had to set one up quickly and on a budget, so in lieu of fancy engineering we set up three rules:
- Any change, whether it's a one-line config update or an entirely new backend, can only go out to a maximum of x% of hosts at a time
- Hosts are taken fully out of rotation before getting an update
- Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions
The first two are easy to enforce programmatically, and the third one relies on the team. While the trade-off is that it's harder to make relatively simple changes, it has served us well so far.
- seanwilson 7y ago> Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions How does that rule work out in practice? If you're doing staged rollouts of new builds because you think there's a decent chance those new builds have bugs in them, shouldn't you realistically expect that rollbacks could have unexpected side-effects?
- tantalor 7y ago> expect that rollbacks could have unexpected Expect the unexpected? Sure anything can happen. But paxys is specifically referring to expected (known) side-effects of rollbacks. For example if Foo and Bar are on versions Foo.3 and Bar.3, and we know rolling back to Bar.2 breaks Foo.3, then you broke backwards compatibility. Generally all combinations of possible rollbacks must continue to work, e.g., {Foo.2, Foo.3} x {Bar.2, Bar.3}. The number of rollbacks is limited to 1 or 2 versions, e.g., Foo.1 and Bar.1 cannot be used anymore.
- gaogao 7y agoThe third one is not too bad to test programmatically, if you write a test harness that spins up a mixed version system.
- hedora 7y agoDowngrade should “just work” (with help from the team) for stateless services, since the old and new have to coexist during upgrade. However, if you’re pushing out new storage systems, supporting downgrade ends up lowering overall reliability in practice. It takes many times more effort to implement and test the downgrade of every disk layout change than it does to ensure that upgrades will always succeed in practice.