3 ms·
With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.
by dx034 8y ago
With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.
- Cthulhu_ 8y agoFor simple PagerDuty alerts I already need 15 minutes to open the app / logs and figure out what's going on.
- wool_gather 8y agoGood point, although the way it's described it sounds like the problem cropped up right after deploy. So they would have been watching it actively. But as said above, 17 minutes to notice, figure out what's going on, decide what to do, and propagate the resolution seems reasonable.
- dfcowell 8y agoNot to mention selectively purging all of the rules created (I assume) at every edge server in their network that were blackholing all of the traffic to the resolver. There's probably a command for it, but all in all 17 minutes seems quite a respectable turnaround time.
- Steltek 8y agoDid they really deploy to all 150 DCs at once? Why was this release not done in phases? Not even a canary?
- dx034 8y agoMaybe they did, the strategy is probably to deploy to small DCs first. But that also means the DDOS detection wouldn't trigger. So probably something that didn't get caught during such tests.