4 ms·

> Paxos is difficult - all of them, Raft, Quorum, leader election, etc. DO NOT USE THEM unless you are Google, Amazon, Walmart, or what not. Even then, do not u

by lucian1900 12y ago

> Paxos is difficult - all of them, Raft, Quorum, leader election, etc. DO NOT USE THEM unless you are Google, Amazon, Walmart, or what not. Even then, do not use them. Instead, I've solved this challenging problem by developing a new Conflict Resolution system Just in case anyone else is considering this: it's a bad idea. Developing a correct consensus algorithm is a lot of hard, unusual work. It's also useless until you can prove it correct. Paxos, Raft and ZAB (Zookeeper's protocol) have already been proven correct. Pick a library for one and use it. Raft is probably easiest to understand and use.

marknadal 12y ago
CRDTs are probably the best thing out there, and the most understandable. They focus on data that does not require complicated leader election algorithms - and this is coming from Facebook and the Cassandra team. Why? Because these systems are very weak and prone toward failure, just check out some of http://aphyr.com/ http://aphyr.com/ 's stuff, he tears a lot of popular databases apart. Even if the theories are "proven right" the implementations are often wrong.
- lucian1900 12y ago
  None of the systems that failed aphyr's tests used a proven consensus algorithm correctly. If you look the Zookeeper test, it performed perfectly, as expected. Even etcd/Consul would pass if they chose to, the defaults are merely bad by choice. CRDTs are useful for salvaging consistency out of an inconsistent database, but they're not the only choice possible.
  - marknadal 12y ago
    Good point. BUT that is a tautology - of course systems that implement it correctly are correct. Unfortunately, there are very few of those things out there... and Zookeeper etcd/Consul are all relatively new, too. This second point goes back to my statement regarding "strong consistency can be built from eventually consistent systems, but you can't go from a strong system to an eventually consistent system." And this is where and why gun makes sense. Start with an eventually consistent system, because it is far far easier and covers 98% of the use cases out there. And only then, when you need consensus on edge-case data, build it out but make sure you understand what you are doing.

marknadal 12y ago

CRDTs are probably the best thing out there, and the most understandable. They focus on data that does not require complicated leader election algorithms - and this is coming from Facebook and the Cassandra team. Why? Because these systems are very weak and prone toward failure, just check out some of http://aphyr.com/ http://aphyr.com/ 's stuff, he tears a lot of popular databases apart. Even if the theories are "proven right" the implementations are often wrong.

lucian1900 12y ago
None of the systems that failed aphyr's tests used a proven consensus algorithm correctly. If you look the Zookeeper test, it performed perfectly, as expected. Even etcd/Consul would pass if they chose to, the defaults are merely bad by choice. CRDTs are useful for salvaging consistency out of an inconsistent database, but they're not the only choice possible.
- marknadal 12y ago
  Good point. BUT that is a tautology - of course systems that implement it correctly are correct. Unfortunately, there are very few of those things out there... and Zookeeper etcd/Consul are all relatively new, too. This second point goes back to my statement regarding "strong consistency can be built from eventually consistent systems, but you can't go from a strong system to an eventually consistent system." And this is where and why gun makes sense. Start with an eventually consistent system, because it is far far easier and covers 98% of the use cases out there. And only then, when you need consensus on edge-case data, build it out but make sure you understand what you are doing.