5 ms·
Googler but nowhere near Gmail, so just educated speculation: * We have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g.
by joatmon-snoo 6y ago
Googler but nowhere near Gmail, so just educated speculation:
* We have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g. roll back a bad flag, quarantine unusual traffic patterns), which means that when something does go wrong it's often a new failure mode that needs custom, specialized mitigation. (e.g. what if you're in a situation where rolling back could make the problem worse? we might be Google, but we don't have magic wands)
* Debugging new failure modes is a coin flip: maybe your existing tools are sufficient to understand what's happening, but if they're not, getting that visibility can in itself be difficult. And just like everyone else, this can become a trial and error process: we find a plausible root cause, design and execute a mitigation based on that understanding, and then get more information that makes very clear that our hypothesis was incomplete (in the worst case, blatantly wrong).
- Aperocky 6y ago> what if you're in a situation where rolling back could make the problem worse? Here comes the poison pills!
- userbinator 6y agoWe have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g. roll back a bad flag, quarantine unusual traffic patterns), which means that when something does go wrong it's often a new failure mode that needs custom, specialized mitigation. As Douglas Adams says, "The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair."
- missblit 6y agoRollback proof bugs are rare, but boy howdy are they exciting. I think I've only seen one so far (unless you count bad data / bad state that persists after a bad change is rolled back... which can also be pretty exciting)
- vitus 6y agoChrome web store has no rollback strategy, there is only roll forward :(
- joshuamorton 6y agoYou can build rollbacks out of rollforwards, although it certainly isn't particularly fun. You patch an update to version N version code so that it's higher than N+1 and roll out the N+2 labelled N.
- Andrex 6y agoIs "exciting" a synonym for "harrowing" where you're from? :P