3 ms·
> Once identified we expedited a rollback which completed in 87 minutes Roll forwards and back should usually be done slowly and carefully. However, during a
by londons_explore 4y ago
> Once identified we expedited a rollback which completed in 87 minutes
Roll forwards and back should usually be done slowly and carefully.
However, during a major outage, there is a good reason to do the rollback very fast. Ie. Within 1 minute. Fast rollbacks end any outage sooner. And if your service is already down, you have nothing to lose and everything to gain.
That provides an interesting infrastructure problem. Fast rollbacks probably involve restarting a large number of servers, having a large number of cold caches, etc. You can find new bugs that only rear their heads during a fast rollback. Things like docker registries getting overloaded, kubernetes control plane being overloaded, other services becoming unhealthy due to a large influx of requests, etc.
The traditional approach is 'dont take risks, don't do a fast rollback'. But that can double or more the length of your downtime. Especially if you roll back the wrong thing first.
I'd therefore encourage preparing ahead of time for fast rollbacks. The dev/test environment should automatically do 'big bang' rollbacks every week. Production rollbacks should even be practiced from time to time, getting faster and faster each time, keeping an eye on metrics to look for instability.
- oefrha 4y ago> And if your service is already down, you have nothing to lose and everything to gain. They had a 5% request failure rate at the peak of this outage. So a botched rollback could definitely do more harm than good if the impact isn’t 100% understood.
- jgrahamc 4y agoExactly.
- londons_explore 4y agoBut there is a good chance that was a 100% request failure rate for a specific service. And when you have 100% of requests failing for a service, you ought to be thinking about doing a fast rollback for that service as long as you have confidence that such a fast rollback won't take other infrastructure down.
- avereveard 4y ago> The impact lasted for almost six hours in total. And faster monitoring, they detected the issue after 4.5 hours and took one to fix
- quickthrower2 4y agoAnother chaos monkey case? Just do a fast rollback every day your D10 rolls a 1.
- vitus 4y agoThere have been YouTube outages that were exacerbated precisely because the SRE teams explicitly trained for the disaster scenario where everything is hard down and needs to be brought back as quickly as possible.