5 ms·
Yes, a Canary lets you limit the damage if some bug sneaks past testing. We've done it for over 10 years, with staged rollouts and automated crash statistics an
by jwatte 10y ago
Yes, a Canary lets you limit the damage if some bug sneaks past testing. We've done it for over 10 years, with staged rollouts and automated crash statistics and such.
The draw back is that prod needs to be tolerant of multiple versions. Which is usually a fine practice in itself, anyway!
- skookum 10y agoThat's not a drawback so much as it's a fact of life. In any large-scale (read: distributed) system trying to provide a high degree of availability, rolling upgrades are the only way code goes out and individual components need to deal with interacting with newer/older dependencies. You can constrain the matrix by only allowing current version plus one back running in production, or forcing deployment orders, and so on, but in modern systems (read: ones where you can't just say "We're taking everything down for 3 hours on Sunday to upgrade.") you can't escape non-atomic upgrades.
- londons_explore 10y agoAnother approach is you start up a full copy of the new system with all new versions, then change loadbalancers to direct all traffic away from the old system and to the new. Then decomission the old system. With dedicated hardware, you need twice as much hardware. With cloud, you only pay double for 10 minutes during the rollout, which usually is very cheap.
- skookum 10y agoThat approach only works if your system is stateless, a caveat which excludes virtually all large-scale systems.