10 ms·
The idea that someone in a partition can get around it by clever routing tricks indicates the author has some “new” definition of partition. Partition isn’t on
by hinkley 2y ago
The idea that someone in a partition can get around it by clever routing tricks indicates the author has some “new” definition of partition.
Partition isn’t one ethernet cable going bad. It’s all the ethernet cables going bad. Redundant network providers for your data center to handle the idiot with the backhoe isn’t surviving partition it’s preventing it in the first place.
- kstrauser 2y agoYep. CAP comes into play when all the mitigations fail. However clever, diligent, and well-executed a network setup, there's some scenario where 2 nodes can't communicate. It's perfectly fine to say we've made it so vastly unlikely that we accept the risk to just take the whole thing offline when that happens. The problem cannot be made to go away. It's not possible.
- wongarsu 2y agoA somewhat plausible scenario that's easy to illustrate would be if you have a DC in the US and one in New Zealand, and all 7 subsea cables connecting New Zealand to the internet go dark. Something like that happened just this year to 13 African nations [1] 1: https://www.techopedia.com/news/when-africa-lost-internet-experts-talk-an-undersea-cable-snafu https://www.techopedia.com/news/when-africa-lost-internet-ex...
- deleted 2y ago[deleted]
- toast0 2y agoYou really do need to consider a full partition, and a partial partition. Both happen. It's most common that you have a full partition when nobody can talk to node A, because it's actually offline. And sometimes you've got a dead uplink for a switch with a couple of nodes behind it that can talk to each other. But partial partitions are really common too. If Node A can talk (bidirectionally) to B and C, but B and C can each only talk to A, you could do something with clever routing tricks. If you have a large enough network, you see this kind of thing all the time, and it's tempting to consider routing tricks. IMHO, it's a lot more realistic to just have a fast feedback between monitoring the network and the people who can go out and clean the ends on the fiber on the switches and routers and what not. The internet as exists is the product of smart people doing routing tricks on a global basis; it's not universally optimal --- you could do better in specific cases all the time, but it's really easy to identify these as a human when something is broken; actually doing probing for host routing is a big exercise for typically marginal benefits. Magic host routing won't help when a backhoe (or directional drill or boat anchor) has determined your redundant fiber paths are all physically present in the same hole though.
- nijave 2y agoI've seen partitions in Postgres clusters where a node gets overloaded to the point it's not responding to health checks and the cluster management software performs a failover. Since the previous master can't be reached but is still running, it accepts writes even after a new replica is promoted. I've seen similar things happen in other software under load as well (which can easily cause cascading failures, too)
- hot_gril 2y agoCould a partition just be the machine hosting one of your DBs going down? When it comes back up, it still has its data, but it's missed out on what happened in the meantime.
- robertlagrant 2y agoNot really - if a database stops you fundamentally don't have availability. CAP is specific to network partitions because it wants to grapple with what happens when clients can talk to nodes in your service, but the nodes can't talk to each other.
- hinkley 2y agoYour question was answered by someone else but you've triggered a different form of dread. We lost a preprod database that held up dev and QA for half a day. To bring the server room up to code they moved the outlets, and when they went to move the power cords, someone thought they could get away with relying on the UPS power during the jiggery. Turned out the UPS that was reporting all green only actually had about 8 seconds of standby power in it, which was a little longer than the IT person needed to untangle the cables. So in the end we were just fucked on a different day than we would have been if we had waited for the next hiccup in the city power. I mention this because if you've split your cluster between three racks and someone manages to power down an entire rack, you're dangerously close to not having quorum and definitely close to not having sufficient throughput for both consumer traffic and rebuilding/resyncing/resilvering. It's a slow motion rock slide that is a cluster that was not sized correctly for user traffic plus recovery traffic as the entire thing limps toward a full on outage when just one more machine needs to be rebooted, because, for instance, your team decided that restart a machine every n/48 hours, to deal with a slow leak is just fine since you're using a consensus protocol that will hide the reboots. Maybe rock slide is the wrong word. It's a train wreck. Once it starts you can't stop it, you can only watch and fret.
- imtringued 2y agoThe thing is, each ethernet cable is a temporal partition. By adding more ethernet cables you're not making partitions less likely, you're making more of them. A distributed system is considered partitioned while the packets are in flight and have not yet reached their destination and come back with a response. This is true even if the system on the other side has not yet failed or the network equipment is functioning properly.
- chronid 2y agoThis is not really correct, and assumes the state of a cloud service (let's say load balancing) is binary. In my experience it's not. The cloud will glitch. The load balancing algo will break subtly for your workload. Your traffic will get blackholed for no apparent reason. I spent a week trying to convince a cloud provider they fucked up (the time it took for them to give us someone who could run the appropriate tcpdump) once. There was no global outage. It's on you to determine if this is important for you or not in your case, but you will need to mitigate it above a certain SLA threshold requirement. It's far for consigning CAP to history or a curiosity, which is like saying you will never have network issues if you use "serverless" stuff. Not talking about anyone in particular, but sometimes I feel people building and using the cloud reach hubris level of surety in their systems - I worked both sides of the fence, and I know the long tails of fuckups that impact customers though...
- nijave 2y agoI remember an article--I think from Gitlab--about random latency and out of order packet issues they were seeing on GCP. Turns out it only happened under a certain amount of load since GCP's routing algorithm uses a different network if traffic is under a certain amount and switches to a more distributed mode when it crosses a threshold. In addition, sometimes you land on a bad piece of hardware but it's not bad enough to trigger the provider's monitoring. Few months ago we had a bad EC2 instance whose network would drop every 2 hours causing a bunch of random errors before recovering for a little while.