3 ms·
> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. In another post
by perlgeek 7y ago
> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes.
In another post mortem by Google I read that Google engineers are trained to roll back recent configuration changes when an outage occurs. Why wasn't this done this time?
- bonestamp2 7y agoMaybe they did rollback and that took longer than the target time (for whatever reason). But it's hard to know since this post mortem is fairly vague.
- gmueckl 7y agoNot just maybe. It says quite clearly that the network was overloaded and, as a result, their configuration changes took too long to arrive at the affected components.
- obstacle1 7y agoThe literal next paragraph: > Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage