In practice, beating the CAP theorem inside the data center is achieved by using a 2-tiered system. Have a set of lightweight controller processes which use a

In practice, beating the CAP theorem inside the data center is achieved by using a 2-tiered system. Have a set of lightweight controller processes which use a consistent majority voting algorithm (Paxos) to replicate the state of the cluster. Since it's lightweight, you can run 5 or 7 of them, meaning 2 or 3 can die. This is enough Availability in practice, per customer input. Have a set of data nodes which actually store the data. Put groups of 3 of these in replica sets and use consistent replication (Paxos). If one or two of them goes down, the controllers reconfigure the replica set, so you get Availability even if 2 of 3 goes down. (Customers love that they get Consistency but 2 of 3 can go down.) This scheme is used in my company's product. The idea comes from academic/Google papers, and is used in other products, too. The scheme does not work for multi-datacenter use-cases. If you want that, you have to give up consistent replication (Paxos) inside the replica sets and use some kind of eventual consistency. This variation of the 2-layered scheme is used by one of the popular competing NoSQL systems.