5 ms·

I'm really waiting for the postmortem. The first services down were networking/datastore, and some minutes later all the others started to fail. My hypothesis i

I'm really waiting for the postmortem. The first services down were networking/datastore, and some minutes later all the others started to fail. My hypothesis is that network failures prevented Paxos, a CP algorithm, to go forward, blocking writes.

tyingq 7y ago
The root cause for the outage in June was bad configuration in the network control plane: https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19...
- solidasparagus 7y ago
  The real root cause was building redundant systems that share a single point of automated failure, creating a massive blast radius. > Thirdly, the software initiating maintenance events had a specific bug, allowing it to deschedule multiple independent software clusters at once, crucially even if those clusters were in different physical locations
  - fulafel 7y ago
    The idea of a single root cause is of course fiction in most failures in complex / redundant systems.