4 ms·
Clean network partitions do happen, although it's certainly not a weekly occurrence. For instance, a metro might be isolated from the rest of the network due t
by vitus 6y ago
Clean network partitions do happen, although it's certainly not a weekly occurrence.
For instance, a metro might be isolated from the rest of the network due to simultaneous fiber cuts.
https://www.bbc.com/news/technology-50851420 https://www.bbc.com/news/technology-50851420
Or we could see heavy, prolonged network congestion on the backbone, impacting comms to/between backend services, while frontend serving is unaffected.
https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19...
(Incidentally, IIRC, Youtube's video serving infra was unaffected, but the backend jobs for youtube.com were largely inaccessible.)
Or improperly-executed maintenance that isolates a metro by draining routers in the wrong order. Insufficiently-provisioned network capacity (or network augments lagging far behind). Backbone traffic engineering gone awry. Underwater fiber cut due to shark attacks. QoS inversion (i.e. user-facing traffic as high priority, but replication as low priority) causing a partition during periods of heavy network load.
While yes, a lot of these scenarios might have (and did!) impact Google's serving capability, Google servers were still IP-reachable. 8.8.8.8 should have still been able to serve DNS (albeit with potentially stale data).
(In more recent news, if you use a transit provider to provide connectivity between your datacenters, and said transit provider suffers a multi-hour outage, that's a clean partition.)