3 ms·

How's that a "dirty little secret"? Raft not supporting asymmetric partitions is somewhere on the third line of the original paper. Besides that, I genuinely w

by r0naa 11y ago

How's that a "dirty little secret"? Raft not supporting asymmetric partitions is somewhere on the third line of the original paper. Besides that, I genuinely wonder how often does this happen in the real world? I see a typical case where parts of your cluster is behind a NAT. Otherwise, what could cause an asymmetric partition?

kelnos 11y ago
That sounds easy, actually. Take a cloud provider like AWS. They divide their regions into "availability zones", which are supposedly in different buildings, don't share the same power, internet connectivity, etc. Say you have 3 AZs, and you put a node in each. A simple network connectivity issue between AZ 1 and 2 (but not between 1 and 3 or 2 and 3) would cause this scenario to happen, assuming I'm understanding this correctly.
vidarh 11y ago
Consider a 3 location network with routes between each pair of locations. Then the link between two of them fails without mechanisms in place to automatically re-route. That's not such an uncommon situation. E.g. quite a few large to mid-sized companies with servers in three locations does not have the in-house skillset to either get BGP set up and/or set up VPNs between the locations and a means to update routes automatically on outages. Some larger shops will have "metro LAN" type setups with separate ports for dedicated connections between racks in different data centres, and won't even have capacity enough on their public ports to handle a failover of traffic normally going on the private connection between two of the data centres over the public port - the expectation is often that the data centre operator will handle redundancy... (until they don't...) It's not a terribly hard thing to fix, but it's also something it seems few people think about until they reach much larger size.

3 ms·

How's that a "dirty little secret"? Raft not supporting asymmetric partitions is somewhere on the third line of the original paper. Besides that, I genuinely w

by r0naa 11y ago

kelnos 11y ago
That sounds easy, actually. Take a cloud provider like AWS. They divide their regions into "availability zones", which are supposedly in different buildings, don't share the same power, internet connectivity, etc. Say you have 3 AZs, and you put a node in each. A simple network connectivity issue between AZ 1 and 2 (but not between 1 and 3 or 2 and 3) would cause this scenario to happen, assuming I'm understanding this correctly.
vidarh 11y ago
Consider a 3 location network with routes between each pair of locations. Then the link between two of them fails without mechanisms in place to automatically re-route. That's not such an uncommon situation. E.g. quite a few large to mid-sized companies with servers in three locations does not have the in-house skillset to either get BGP set up and/or set up VPNs between the locations and a means to update routes automatically on outages. Some larger shops will have "metro LAN" type setups with separate ports for dedicated connections between racks in different data centres, and won't even have capacity enough on their public ports to handle a failover of traffic normally going on the private connection between two of the data centres over the public port - the expectation is often that the data centre operator will handle redundancy... (until they don't...) It's not a terribly hard thing to fix, but it's also something it seems few people think about until they reach much larger size.