3 ms·
> On October 27th at 14:00, one day before the outage, we enabled this feature on a backend service that is responsible for traffic routing. As part of this rol
by wizwit999 5y ago
> On October 27th at 14:00, one day before the outage, we enabled this feature on a backend service that is responsible for traffic routing. As part of this rollout, in order to prepare for the increased traffic we typically see at the end of the year, we also increased the number of nodes supporting traffic routing by 50%.
Seems like the smoking gun, this should have been identified and rolled back much earlier.
- yuliyp 5y agoIt's obvious when it's pointed out in an article like this. It's less clear when it's one of many changes that could have been happening in a day, and it was an operation that was considered "safe" given that it had been done multiple times for other services in the preceding months.
- Karrot_Kream 5y agoIf reading a postmortem makes the smoking gun obvious, then the postmortem is doing its job. Don't mistake the amount of investigation that goes into a postmortem for the available information and mental headspace during an outage.
- wizwit999 5y agoI've been in my fair share of incidents so I'm aware of how they work. But they knew it was an issue related to Consul within hours. It shouldn't take more than two days before they check for recent deployments made to Consul.
- orangepenguin 5y agoIn my experience, there's typically more than one "smoking gun". The problem isn't finding one, it's eliminating all of the "smoking guns" that aren't actually related to the outage. If I worked at an organization with many teams deploying updates multiple times per day and several same day events seemed related, I would probably also put less weight on a gradual, months-long deployment that had completed a day prior.