6 ms·

Isn't managing consensus extremely hard to do? Wouldn't one want to rely on a proven solution rather than spinning up a new solution?

empath75 7y ago
Well, there's etcd, consul, etc.. edit: i guess i should RTFA -- looks like they are going to build raft into kafka -- at this point, I'd think raft libraries are fairly mature.
james-mcelwain 7y ago
There are a number of Java libraries that implement Raft already -- it's a fairly well understood algorithm.
- rad_gruchalski 7y ago
  TIL: there is an Apache Incubator raft project: http://ratis.incubator.apache.org/ http://ratis.incubator.apache.org/.
Plugawy 7y ago
The document mentions using Raft for consensus and coordination - it's the same approach used by Etcd, Consul, Serf, RethinkDB and other systems. AFAIK it's easier to implement and understand than Zookeeper's consensus solution. https://raft.github.io https://raft.github.io
- deleted 7y ago
  [deleted]
- ypcx 7y ago
  Kafka was thinking about etcd: https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+... Considering most Kafkas will probably run in Kubernetes at some point, they could have shared the etcd used by Kubernetes.
  - Plugawy 7y ago
    Sounds like a bad idea, no? Last ting you want is whole cluster going down because Kafka is misbehaving
    - ypcx 7y ago
      I don't know. First, there may be some separation achievable - but I'm just guessing. Second, if Kafka is your primary workload, you don't want it to misbehave in any case. But of course I get your point. I'm just saying I've seen people thinking about it that way in some Github Issues.
  - edude03 7y ago
    There is also Zetcd[0] if you want to use etcd and kafka together today. We use it right now and it seems to work [0] https://coreos.com/blog/introducing-zetcd https://coreos.com/blog/introducing-zetcd
leetbulb 7y ago
There are other proven solutions that can be embedded such as Raft.
chrsig 7y ago
Yes, on both counts. However, requiring zookeeper as a dependency is also a barrier to adoption and increases the operational surface area, so offloading consensus management to zookeeper is not without trade offs.
binarymax 7y ago
My thoughts exactly. I don't work with Kafka, but I do heavy work with Solr that also has ZK as a dependency. To end users of whatever platform it is keeping together, Zookeeper is often seen as an unwanted dependency. The unwanted sentiment arises because maintaining zookeeper is not a gimme, and it takes additional knowledge and overhead to maintain. You need a quorum to be kept alive with an odd number (greater than one) of instances. So the question often arises, "how can we get rid of zookeeper?".
minxomat 7y ago
Not to mention that ZooKeeper passed Jepsen (on the first try IIRC). Don't replace something so extraordinarily dependable just because it's a dependency.
- rad_gruchalski 7y ago
  Zookeeper can also be embedded.
  - justnoise 7y ago
    As can etcd. Edit: Oh right, the fact that etcd is golang might make that an issue for Kafka...
helper 7y ago
As an operator, having fewer dependencies I have to deploy and worry about is a boon to my productivity. If you were building a kafka like system in house for private use only, then yes I agree with your sentiment. If you are building something to be used by hundreds or thousands of organizations then the cost benefit tradeoffs shift to where it probably makes sense to pull the consensus logic into the primary application itself.
- cafxx 7y ago
  As an operator, shouldn't you worry if something battle-tested, that implements extremely critical functions that are tricky to implement correctly, is replaced with something that is unproven? As someone who enjoys sleeping at night, I wouldn't poke something like that with a stick before it has matured for a few years, and before many brave (?) operators have smoothed out most rough edges by landing on them repeatedly with their faces.
  - helper 7y ago
    Yeah, don't deploy it when its in beta or its first few releases. At some point it will be stable and safe at which point everyone from then on will benefit.
- agallego 7y ago
  Note that you still have the same number of fault domains if you read the actual proposal. It does not simplify the system. It is simply replacing the ZK ensemble w/ its own raft impl. Same number of JVMs, same operational complexity.
  - helper 7y ago
    Thanks for pointing that out. Thats what I get for not actually reading the proposal.
mangatmodi 7y ago
Finally somebody said it. Yes it is hard. Zookeeper is time tested and I don't think there is anything revolutionary to keep state less than 2n nodes. Either Kafka will compromise on consistency or give a solution which is as performant as Zookeeper. May be they are looking for something which is more specific to Kafka and can be done in a much easier algorithm than Zab?
czinck 7y ago
Reading the proposal, it seems like they want to replace it because the ZooKeeper API is causing some issues for them, so even if ZooKeeper has perfect consensus, Kafka itself won't necessarily. From the "Metadata as an Event Log" section > although ZooKeeper is the store of record, the state in ZooKeeper often doesn't match the state that is held in memory in the controller Maybe it would be better to put the effort in to changing ZooKeeper instead of writing their own consensus system in Kafka, but I would assume they know that tradeoff better than us since they work so closely with ZooKeeper.
- rad_gruchalski 7y ago
  I do find that statement really confusing. That implies they could have errors in the code handling the consensus? Hard to believe but otoh, they’re not really saying anything about zookeeper causing problems. Maybe the correct solution is to understand why the states don’t match in the first place?
atombender 7y ago
No, contrary to the sibling comments I would disagree that consensus is difficult. The theory can be somewhat daunting, and ZooKeeper's consensus protocol is complex. The most advanced consensus protocol today, Paxos (and its variations), is notoriously hard to understand. But then we got the Raft paper, which has single-handedly essentially managed to commoditize distributed consensus. Raft is an elegant and simple protocol that has been implemented in many languages. To implement primitives like leader election and consistent, distributed reads/writes, you can grab an off-the-shelf library like Hashicorp's Raft library, which does all the heavy lifting as long as you implement the state storage yourself. It's absurdly trivial to turn a single-node app into a replicated, fully redundant one. Of course, a project like Kafka might have particular requirements (I wouldn't know) that would mean Raft would not be a suitable solution for them, or would require modifications to the protocol. CockroachDB, for example, had to modify Raft ("MultiRaft") in order to achieve the scalability they needed within Raft's paradigm. Raft is a starting point more than a finished solution.
- necubi 7y ago
  I'd read through some of Aphyr's jepsen tests [0] if you think implementing consensus algorithms correctly is easy. Nearly every distributed database he's looked at has consistency bugs (with the notable exception of zookeeper). [0] https://jepsen.io/analyses https://jepsen.io/analyses
  - atombender 7y ago
    I've read every single one of Aphyr's tests. Aphyr's analyses probe a broader range of distributed consistency problems in replicated stateful databases. Consensus is a necessary mechanism to achieve distributed consistency, but if you look at the problems highlighted in products like CockroachDB and TiDB, they relate to secondary effects of failures to ensure linearizability of reads and writes. You can have a perfectly functioning consensus algorithm and still battle inconsistent reads across nodes, loss of data on node failure or network splits, write skew, and so on. For example, TiKV uses Raft, yet the problems uncovered in TiKV have nothing to do with Raft itself. Conversely, those found in Elasticsearch are absolutely caused by its consensus algorithm, Zen, which was invented in ignorance of modern consensus research (Paxos, etc.). If you read my comment carefully, you'll note that it's above all an endorsement of Raft, which is a milestone in that it democratizes advanced consensus and makes it a tool anyone can use. My point isn't that anyone can trivially whip up an infinitely scalable, consistent database because consensus is a solved problem, but that we have solid foundational theory to build on, to the point where it is a well-understood problem (albeit with many complicated implementations) and not black magic. Also, I did not use the term "easy". But I also think the OP's phrasing "extremely hard" is not true, unless your requirements are extremely hard – which, again, may be the case with Kafka; scalability tends to be diametrically opposed to consistency.
RantyDave 7y ago
Exactly, and tbh this kind of thing pisses me off. The "Kafka people" seem to imagine that what Zookeeper does is dead simple and they can bash together their own in a weekend and it'll be better. Zookeeper is great. Weird, for sure, and a bit of a pain in the arse. But it will neither lie to you nor lose data and this is much, much harder to achieve than you might think.

empath75 7y ago

Well, there's etcd, consul, etc.. edit: i guess i should RTFA -- looks like they are going to build raft into kafka -- at this point, I'd think raft libraries are fairly mature.

james-mcelwain 7y ago

There are a number of Java libraries that implement Raft already -- it's a fairly well understood algorithm.

rad_gruchalski 7y ago
TIL: there is an Apache Incubator raft project: http://ratis.incubator.apache.org/ http://ratis.incubator.apache.org/.

Plugawy 7y ago

The document mentions using Raft for consensus and coordination - it's the same approach used by Etcd, Consul, Serf, RethinkDB and other systems. AFAIK it's easier to implement and understand than Zookeeper's consensus solution. https://raft.github.io https://raft.github.io

deleted 7y ago
[deleted]
ypcx 7y ago
Kafka was thinking about etcd: https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+... Considering most Kafkas will probably run in Kubernetes at some point, they could have shared the etcd used by Kubernetes.
- Plugawy 7y ago
  Sounds like a bad idea, no? Last ting you want is whole cluster going down because Kafka is misbehaving
  - ypcx 7y ago
    I don't know. First, there may be some separation achievable - but I'm just guessing. Second, if Kafka is your primary workload, you don't want it to misbehave in any case. But of course I get your point. I'm just saying I've seen people thinking about it that way in some Github Issues.
- edude03 7y ago
  There is also Zetcd[0] if you want to use etcd and kafka together today. We use it right now and it seems to work [0] https://coreos.com/blog/introducing-zetcd https://coreos.com/blog/introducing-zetcd

leetbulb 7y ago

There are other proven solutions that can be embedded such as Raft.

chrsig 7y ago

Yes, on both counts. However, requiring zookeeper as a dependency is also a barrier to adoption and increases the operational surface area, so offloading consensus management to zookeeper is not without trade offs.

binarymax 7y ago

My thoughts exactly. I don't work with Kafka, but I do heavy work with Solr that also has ZK as a dependency. To end users of whatever platform it is keeping together, Zookeeper is often seen as an unwanted dependency. The unwanted sentiment arises because maintaining zookeeper is not a gimme, and it takes additional knowledge and overhead to maintain. You need a quorum to be kept alive with an odd number (greater than one) of instances. So the question often arises, "how can we get rid of zookeeper?".

minxomat 7y ago

Not to mention that ZooKeeper passed Jepsen (on the first try IIRC). Don't replace something so extraordinarily dependable just because it's a dependency.

rad_gruchalski 7y ago
Zookeeper can also be embedded.
- justnoise 7y ago
  As can etcd. Edit: Oh right, the fact that etcd is golang might make that an issue for Kafka...

helper 7y ago

As an operator, having fewer dependencies I have to deploy and worry about is a boon to my productivity. If you were building a kafka like system in house for private use only, then yes I agree with your sentiment. If you are building something to be used by hundreds or thousands of organizations then the cost benefit tradeoffs shift to where it probably makes sense to pull the consensus logic into the primary application itself.

cafxx 7y ago
As an operator, shouldn't you worry if something battle-tested, that implements extremely critical functions that are tricky to implement correctly, is replaced with something that is unproven? As someone who enjoys sleeping at night, I wouldn't poke something like that with a stick before it has matured for a few years, and before many brave (?) operators have smoothed out most rough edges by landing on them repeatedly with their faces.
- helper 7y ago
  Yeah, don't deploy it when its in beta or its first few releases. At some point it will be stable and safe at which point everyone from then on will benefit.
agallego 7y ago
Note that you still have the same number of fault domains if you read the actual proposal. It does not simplify the system. It is simply replacing the ZK ensemble w/ its own raft impl. Same number of JVMs, same operational complexity.
- helper 7y ago
  Thanks for pointing that out. Thats what I get for not actually reading the proposal.

mangatmodi 7y ago

Finally somebody said it. Yes it is hard. Zookeeper is time tested and I don't think there is anything revolutionary to keep state less than 2n nodes. Either Kafka will compromise on consistency or give a solution which is as performant as Zookeeper. May be they are looking for something which is more specific to Kafka and can be done in a much easier algorithm than Zab?

czinck 7y ago

Reading the proposal, it seems like they want to replace it because the ZooKeeper API is causing some issues for them, so even if ZooKeeper has perfect consensus, Kafka itself won't necessarily. From the "Metadata as an Event Log" section > although ZooKeeper is the store of record, the state in ZooKeeper often doesn't match the state that is held in memory in the controller Maybe it would be better to put the effort in to changing ZooKeeper instead of writing their own consensus system in Kafka, but I would assume they know that tradeoff better than us since they work so closely with ZooKeeper.

rad_gruchalski 7y ago
I do find that statement really confusing. That implies they could have errors in the code handling the consensus? Hard to believe but otoh, they’re not really saying anything about zookeeper causing problems. Maybe the correct solution is to understand why the states don’t match in the first place?

atombender 7y ago

No, contrary to the sibling comments I would disagree that consensus is difficult. The theory can be somewhat daunting, and ZooKeeper's consensus protocol is complex. The most advanced consensus protocol today, Paxos (and its variations), is notoriously hard to understand. But then we got the Raft paper, which has single-handedly essentially managed to commoditize distributed consensus. Raft is an elegant and simple protocol that has been implemented in many languages. To implement primitives like leader election and consistent, distributed reads/writes, you can grab an off-the-shelf library like Hashicorp's Raft library, which does all the heavy lifting as long as you implement the state storage yourself. It's absurdly trivial to turn a single-node app into a replicated, fully redundant one. Of course, a project like Kafka might have particular requirements (I wouldn't know) that would mean Raft would not be a suitable solution for them, or would require modifications to the protocol. CockroachDB, for example, had to modify Raft ("MultiRaft") in order to achieve the scalability they needed within Raft's paradigm. Raft is a starting point more than a finished solution.

necubi 7y ago
I'd read through some of Aphyr's jepsen tests [0] if you think implementing consensus algorithms correctly is easy. Nearly every distributed database he's looked at has consistency bugs (with the notable exception of zookeeper). [0] https://jepsen.io/analyses https://jepsen.io/analyses
- atombender 7y ago
  I've read every single one of Aphyr's tests. Aphyr's analyses probe a broader range of distributed consistency problems in replicated stateful databases. Consensus is a necessary mechanism to achieve distributed consistency, but if you look at the problems highlighted in products like CockroachDB and TiDB, they relate to secondary effects of failures to ensure linearizability of reads and writes. You can have a perfectly functioning consensus algorithm and still battle inconsistent reads across nodes, loss of data on node failure or network splits, write skew, and so on. For example, TiKV uses Raft, yet the problems uncovered in TiKV have nothing to do with Raft itself. Conversely, those found in Elasticsearch are absolutely caused by its consensus algorithm, Zen, which was invented in ignorance of modern consensus research (Paxos, etc.). If you read my comment carefully, you'll note that it's above all an endorsement of Raft, which is a milestone in that it democratizes advanced consensus and makes it a tool anyone can use. My point isn't that anyone can trivially whip up an infinitely scalable, consistent database because consensus is a solved problem, but that we have solid foundational theory to build on, to the point where it is a well-understood problem (albeit with many complicated implementations) and not black magic. Also, I did not use the term "easy". But I also think the OP's phrasing "extremely hard" is not true, unless your requirements are extremely hard – which, again, may be the case with Kafka; scalability tends to be diametrically opposed to consistency.

RantyDave 7y ago

Exactly, and tbh this kind of thing pisses me off. The "Kafka people" seem to imagine that what Zookeeper does is dead simple and they can bash together their own in a weekend and it'll be better. Zookeeper is great. Weird, for sure, and a bit of a pain in the arse. But it will neither lie to you nor lose data and this is much, much harder to achieve than you might think.