4 ms·
You really do need to consider a full partition, and a partial partition. Both happen. It's most common that you have a full partition when nobody can talk to
by toast0 2y ago
You really do need to consider a full partition, and a partial partition. Both happen.
It's most common that you have a full partition when nobody can talk to node A, because it's actually offline. And sometimes you've got a dead uplink for a switch with a couple of nodes behind it that can talk to each other.
But partial partitions are really common too. If Node A can talk (bidirectionally) to B and C, but B and C can each only talk to A, you could do something with clever routing tricks. If you have a large enough network, you see this kind of thing all the time, and it's tempting to consider routing tricks. IMHO, it's a lot more realistic to just have a fast feedback between monitoring the network and the people who can go out and clean the ends on the fiber on the switches and routers and what not. The internet as exists is the product of smart people doing routing tricks on a global basis; it's not universally optimal --- you could do better in specific cases all the time, but it's really easy to identify these as a human when something is broken; actually doing probing for host routing is a big exercise for typically marginal benefits. Magic host routing won't help when a backhoe (or directional drill or boat anchor) has determined your redundant fiber paths are all physically present in the same hole though.
- nijave 2y agoI've seen partitions in Postgres clusters where a node gets overloaded to the point it's not responding to health checks and the cluster management software performs a failover. Since the previous master can't be reached but is still running, it accepts writes even after a new replica is promoted. I've seen similar things happen in other software under load as well (which can easily cause cascading failures, too)