4 ms·

Interesting story, the Amazon datacenters were built in such a way they were reliant on WAN links between them to act as one. So take your average datacenter,

Interesting story, the Amazon datacenters were built in such a way they were reliant on WAN links between them to act as one. So take your average datacenter, let's say it has 60,000 machines, and now you put it in to 3 datacenters of 20,000 machines each. For reliability reasons. (nevermind the math or common sense) This is what amazon did. And the single fiber link between the datacenters had a tendency to get dug up by everyone pretty much. During these outages each DC had its external connectivity working. And now you know why Dynamo was built the way it was. Full split brain mode was common because service owners would put 50% of their fleet in 2 datacenters. Usually EXACTLY 50% as well, since humans tend to "snap to" even numbers. One last point, in the CAP theorem proof, "partitions" really just means loss of packets to/from one node. That is the smallest/general case. Even if one can handle single node failure well (eg: dynamo systems) it may or may not be able to handle full split-brain mode. That is a lot harder and there isn't any real CS to help you here (not that the CAP theorem is helpful)

aphyr 13y ago
One last point, in the CAP theorem proof, "partitions" really just means loss of packets to/from one node. That is the smallest/general case. I don't know where you got this idea from. The formal model in Gilbert and Lynch's proof is arbitrary packet loss, which is equivalent to an arbitrary pattern of isolation between groups (of any size) of nodes. See section 2.3: http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pd... Even if one can handle single node failure well (eg: dynamo systems) it may or may not be able to handle full split-brain mode. Just so we're clear, Dynamo does handle totally isolated components during partition. This is an explicit goal of the paper, and it provides specific guarantees about availability in those circumstances. That is a lot harder and there isn't any real CS to help you here (not that the CAP theorem is helpful) There is a great deal of CS to help you deal with partitions. You might start with Lamport, Lynch, Gray, Brewer, Chandra, Cheriton & Skeen, etc...

aphyr 13y ago

One last point, in the CAP theorem proof, "partitions" really just means loss of packets to/from one node. That is the smallest/general case. I don't know where you got this idea from. The formal model in Gilbert and Lynch's proof is arbitrary packet loss, which is equivalent to an arbitrary pattern of isolation between groups (of any size) of nodes. See section 2.3: http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pd... Even if one can handle single node failure well (eg: dynamo systems) it may or may not be able to handle full split-brain mode. Just so we're clear, Dynamo does handle totally isolated components during partition. This is an explicit goal of the paper, and it provides specific guarantees about availability in those circumstances. That is a lot harder and there isn't any real CS to help you here (not that the CAP theorem is helpful) There is a great deal of CS to help you deal with partitions. You might start with Lamport, Lynch, Gray, Brewer, Chandra, Cheriton & Skeen, etc...