4 ms·

Agreed, network errors are very common but the partial failure scenario, with a fully functional client-server connection but failed server-server replication c

Agreed, network errors are very common but the partial failure scenario, with a fully functional client-server connection but failed server-server replication connection, seems rare. Maybe I don't read the network failure post-mortems carefully enough. When routers fail or are misconfigured, all hell breaks loose rather than clean partitions occurring.

tonyarkles 6y ago
One place I’ve run into these kinds of errors frequently was in a heavily locked down corporate LAN. All kinds of messed up firewall rules doing wild things. As an example, had a node with a misconfigured egress rule. Trying to connect to it, the node would receive the SYN, but the SYN/ACK would get eaten on egress... so for 60 seconds both sides believed a connection existed, but you couldn’t do anything. Then it times out and starts all over again.
lclarkmichalek 6y ago
The classic would be a geographically distributed datastore with intra-datacenter link being fine, but inter-datacenter being down/degraded. Clients within the datacenter would still be able to talk to the server fine, but server to server inter-datacenter is not going to be OK. Another part of it is that all hell breaking loose can sometimes look very similar to clean partitions, depending on the network. When network QoS classes are available, what appears to be increased latency or packet loss for one class may be 100% packet loss for the other. Though, you're totally right that all hell breaking loose can be very different - the set of systems that handle flapping & other ugly connection failures well is smaller than the set of systems that handle clean disconnections well :)
vitus 6y ago
Clean network partitions do happen, although it's certainly not a weekly occurrence. For instance, a metro might be isolated from the rest of the network due to simultaneous fiber cuts. https://www.bbc.com/news/technology-50851420 https://www.bbc.com/news/technology-50851420 Or we could see heavy, prolonged network congestion on the backbone, impacting comms to/between backend services, while frontend serving is unaffected. https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19... (Incidentally, IIRC, Youtube's video serving infra was unaffected, but the backend jobs for youtube.com were largely inaccessible.) Or improperly-executed maintenance that isolates a metro by draining routers in the wrong order. Insufficiently-provisioned network capacity (or network augments lagging far behind). Backbone traffic engineering gone awry. Underwater fiber cut due to shark attacks. QoS inversion (i.e. user-facing traffic as high priority, but replication as low priority) causing a partition during periods of heavy network load. While yes, a lot of these scenarios might have (and did!) impact Google's serving capability, Google servers were still IP-reachable. 8.8.8.8 should have still been able to serve DNS (albeit with potentially stale data). (In more recent news, if you use a transit provider to provide connectivity between your datacenters, and said transit provider suffers a multi-hour outage, that's a clean partition.)
cthalupa 6y ago
>partial failure scenario, with a fully functional client-server connection but failed server-server replication connection, seems rare You're more likely to see it when you are running services across multiple datacenters/availability zones/regions/whatever. It still happens frequently enough that it's a scenario worth keeping in mind. You probably have your application servers local to your database servers in that other datacenter, so if your public facing network falls over in a datacenter, there's probably still some amount of work happening that modifies things in the database - what happens when that connectivity is restored? Being able to gracefully handle network partitions is hugely important in distributed computing.

tonyarkles 6y ago

One place I’ve run into these kinds of errors frequently was in a heavily locked down corporate LAN. All kinds of messed up firewall rules doing wild things. As an example, had a node with a misconfigured egress rule. Trying to connect to it, the node would receive the SYN, but the SYN/ACK would get eaten on egress... so for 60 seconds both sides believed a connection existed, but you couldn’t do anything. Then it times out and starts all over again.

lclarkmichalek 6y ago

The classic would be a geographically distributed datastore with intra-datacenter link being fine, but inter-datacenter being down/degraded. Clients within the datacenter would still be able to talk to the server fine, but server to server inter-datacenter is not going to be OK. Another part of it is that all hell breaking loose can sometimes look very similar to clean partitions, depending on the network. When network QoS classes are available, what appears to be increased latency or packet loss for one class may be 100% packet loss for the other. Though, you're totally right that all hell breaking loose can be very different - the set of systems that handle flapping & other ugly connection failures well is smaller than the set of systems that handle clean disconnections well :)

vitus 6y ago

Clean network partitions do happen, although it's certainly not a weekly occurrence. For instance, a metro might be isolated from the rest of the network due to simultaneous fiber cuts. https://www.bbc.com/news/technology-50851420 https://www.bbc.com/news/technology-50851420 Or we could see heavy, prolonged network congestion on the backbone, impacting comms to/between backend services, while frontend serving is unaffected. https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19... (Incidentally, IIRC, Youtube's video serving infra was unaffected, but the backend jobs for youtube.com were largely inaccessible.) Or improperly-executed maintenance that isolates a metro by draining routers in the wrong order. Insufficiently-provisioned network capacity (or network augments lagging far behind). Backbone traffic engineering gone awry. Underwater fiber cut due to shark attacks. QoS inversion (i.e. user-facing traffic as high priority, but replication as low priority) causing a partition during periods of heavy network load. While yes, a lot of these scenarios might have (and did!) impact Google's serving capability, Google servers were still IP-reachable. 8.8.8.8 should have still been able to serve DNS (albeit with potentially stale data). (In more recent news, if you use a transit provider to provide connectivity between your datacenters, and said transit provider suffers a multi-hour outage, that's a clean partition.)

cthalupa 6y ago

>partial failure scenario, with a fully functional client-server connection but failed server-server replication connection, seems rare You're more likely to see it when you are running services across multiple datacenters/availability zones/regions/whatever. It still happens frequently enough that it's a scenario worth keeping in mind. You probably have your application servers local to your database servers in that other datacenter, so if your public facing network falls over in a datacenter, there's probably still some amount of work happening that modifies things in the database - what happens when that connectivity is restored? Being able to gracefully handle network partitions is hugely important in distributed computing.