3 ms·

No, contrary to the sibling comments I would disagree that consensus is difficult. The theory can be somewhat daunting, and ZooKeeper's consensus protocol is co

No, contrary to the sibling comments I would disagree that consensus is difficult. The theory can be somewhat daunting, and ZooKeeper's consensus protocol is complex. The most advanced consensus protocol today, Paxos (and its variations), is notoriously hard to understand. But then we got the Raft paper, which has single-handedly essentially managed to commoditize distributed consensus. Raft is an elegant and simple protocol that has been implemented in many languages. To implement primitives like leader election and consistent, distributed reads/writes, you can grab an off-the-shelf library like Hashicorp's Raft library, which does all the heavy lifting as long as you implement the state storage yourself. It's absurdly trivial to turn a single-node app into a replicated, fully redundant one. Of course, a project like Kafka might have particular requirements (I wouldn't know) that would mean Raft would not be a suitable solution for them, or would require modifications to the protocol. CockroachDB, for example, had to modify Raft ("MultiRaft") in order to achieve the scalability they needed within Raft's paradigm. Raft is a starting point more than a finished solution.

necubi 7y ago
I'd read through some of Aphyr's jepsen tests [0] if you think implementing consensus algorithms correctly is easy. Nearly every distributed database he's looked at has consistency bugs (with the notable exception of zookeeper). [0] https://jepsen.io/analyses https://jepsen.io/analyses
- atombender 7y ago
  I've read every single one of Aphyr's tests. Aphyr's analyses probe a broader range of distributed consistency problems in replicated stateful databases. Consensus is a necessary mechanism to achieve distributed consistency, but if you look at the problems highlighted in products like CockroachDB and TiDB, they relate to secondary effects of failures to ensure linearizability of reads and writes. You can have a perfectly functioning consensus algorithm and still battle inconsistent reads across nodes, loss of data on node failure or network splits, write skew, and so on. For example, TiKV uses Raft, yet the problems uncovered in TiKV have nothing to do with Raft itself. Conversely, those found in Elasticsearch are absolutely caused by its consensus algorithm, Zen, which was invented in ignorance of modern consensus research (Paxos, etc.). If you read my comment carefully, you'll note that it's above all an endorsement of Raft, which is a milestone in that it democratizes advanced consensus and makes it a tool anyone can use. My point isn't that anyone can trivially whip up an infinitely scalable, consistent database because consensus is a solved problem, but that we have solid foundational theory to build on, to the point where it is a well-understood problem (albeit with many complicated implementations) and not black magic. Also, I did not use the term "easy". But I also think the OP's phrasing "extremely hard" is not true, unless your requirements are extremely hard – which, again, may be the case with Kafka; scalability tends to be diametrically opposed to consistency.
hinkley 7y ago
Raft is not going to be the last consensus protocol. If that assertion is true, then the replacement will exist because something in Raft is difficult or impossible to do. In which case consensus is hard, for some quantity of hard. The one that's most obvious to me is that Raft violates one of the 8 Fallacies of Network Computing: the network is homogeneous. The star topology of messages means that some messages - identical messages with different destinations - will be competing for bandwidth. It also means electing the slowest machine on the longest network link is probably a bad idea. I've certainly heard of this sort of problem happening.

necubi 7y ago

I'd read through some of Aphyr's jepsen tests [0] if you think implementing consensus algorithms correctly is easy. Nearly every distributed database he's looked at has consistency bugs (with the notable exception of zookeeper). [0] https://jepsen.io/analyses https://jepsen.io/analyses

atombender 7y ago
I've read every single one of Aphyr's tests. Aphyr's analyses probe a broader range of distributed consistency problems in replicated stateful databases. Consensus is a necessary mechanism to achieve distributed consistency, but if you look at the problems highlighted in products like CockroachDB and TiDB, they relate to secondary effects of failures to ensure linearizability of reads and writes. You can have a perfectly functioning consensus algorithm and still battle inconsistent reads across nodes, loss of data on node failure or network splits, write skew, and so on. For example, TiKV uses Raft, yet the problems uncovered in TiKV have nothing to do with Raft itself. Conversely, those found in Elasticsearch are absolutely caused by its consensus algorithm, Zen, which was invented in ignorance of modern consensus research (Paxos, etc.). If you read my comment carefully, you'll note that it's above all an endorsement of Raft, which is a milestone in that it democratizes advanced consensus and makes it a tool anyone can use. My point isn't that anyone can trivially whip up an infinitely scalable, consistent database because consensus is a solved problem, but that we have solid foundational theory to build on, to the point where it is a well-understood problem (albeit with many complicated implementations) and not black magic. Also, I did not use the term "easy". But I also think the OP's phrasing "extremely hard" is not true, unless your requirements are extremely hard – which, again, may be the case with Kafka; scalability tends to be diametrically opposed to consistency.

hinkley 7y ago

Raft is not going to be the last consensus protocol. If that assertion is true, then the replacement will exist because something in Raft is difficult or impossible to do. In which case consensus is hard, for some quantity of hard. The one that's most obvious to me is that Raft violates one of the 8 Fallacies of Network Computing: the network is homogeneous. The star topology of messages means that some messages - identical messages with different destinations - will be competing for bandwidth. It also means electing the slowest machine on the longest network link is probably a bad idea. I've certainly heard of this sort of problem happening.