3 ms·
I really want to see some comments from SREs (or similar) at major companies on this. It seems like a lot have a rollback first, ask questions later policy, and
by joshuamorton 10y ago
I really want to see some comments from SREs (or similar) at major companies on this. It seems like a lot have a rollback first, ask questions later policy, and I'd love to hear about how they mitigate or flat out avoid the problems described in this article, or if I'm just mistaken.
- morgante 10y agoA lot of them rely heavily on feature flags and gradual rollouts.
- cavisne 10y agoI think the burden is really on the software engineer here, any change that goes out needs to be forwards and backwards compatible. A good pattern ive seen for this is having 2 stages to your test environment, a single host and then the rest of the fleet. First you deploy to the single host. Then you run "compatibility" tests that essentially write to the new code, and read from the old code and vice versa. Thats the ideal way, but also a lot of work. A simpler way it just to roll out changes very slowly, and hopefully detect a deployment with compatibility issues before it effects too many customers.
- cobbzilla 10y agoIt's also easier when you have a massive fleet of machines, spread across multiple geographic regions. Combine feature toggles + backwards/forwards compatibility + decent error reporting, and deployments can become more automated. Start by rolling out to 1% of machines in one region. Let it run for a while. No errors? escalate to more machines, or more regions, etc. Release frequency and propagation speed across the network can be as fast or slow as you are comfortable with.
- peterwwillis 10y agoSREs only have a supporting role to play in change rollback. They maintain the tools/services used by SWEs, DBAs, etc to follow the rollback procedure documented in Change Management. Rollback is often a suitable first response because the rollback is not a button, it is a procedure, and it can be practiced in a test environment and on a small subset of production without significant customer impact. Like the other commenter suggested, gradual rollout is a great way to spot bugs or customer feedback. You can accomplish this by building deployment tools that allow for targeted deployment, and by maintaining services that can be targeted by release. For example, you can use your Configuration Management engine to snapshot a DB, deploy schema changes to it and then your code to the app servers using only that DB. If there's a problem you can roll back the code, the data, the schema, whatever changed at that release version, and reload all the services. For things that affect customer assets you'll need extra procedures to work around potential failures there, but that's not complicated (usually).
- svachalek 10y agoThe faster you roll back, the less complications there are to deal with. Just think about all the damage that's accumulating if you've got millions of users hitting the site while you're figuring out a way forward. So, as they say, you roll stuff out very slowly and roll it back quickly. Turning off feature flags as the article suggests is IMO just a different implementation of rolling back. Now you run the old code again and have all the same risks as an actual rollback (just more self-contained). Edit: note that you also have to take into account how long it takes to notice the problem. If you're rolling out and notice something bad when you hit 1% or 10% of users, it's one thing, but if you hit 100% rollout and notice bad things happening a week later it's a different situation. For one thing, hopefully this indicates the problem is not spectacularly fatal, and for another you've already got users used to the new behaviors and who knows how much accumulated damage. So then it's worthwhile to step back a bit and figure out how to recover.
- boulos 10y agoI'm not in SRE but I'll say what I've seen us do for Compute Engine: every component is testing for roll forward and roll back as part of qualification in staging. This makes it more hassle when you write the code (first add your new field, then rely on it only once it definitely won't be rolled back), but we absolutely use our "rollback button" frequently.