7 ms·

KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

139 points by telekid 7y ago

mykowebhn 7y ago
Isn't managing consensus extremely hard to do? Wouldn't one want to rely on a proven solution rather than spinning up a new solution?
- empath75 7y ago
  Well, there's etcd, consul, etc.. edit: i guess i should RTFA -- looks like they are going to build raft into kafka -- at this point, I'd think raft libraries are fairly mature.
- james-mcelwain 7y ago
  There are a number of Java libraries that implement Raft already -- it's a fairly well understood algorithm.
  - rad_gruchalski 7y ago
    TIL: there is an Apache Incubator raft project: http://ratis.incubator.apache.org/ http://ratis.incubator.apache.org/.
- Plugawy 7y ago
  The document mentions using Raft for consensus and coordination - it's the same approach used by Etcd, Consul, Serf, RethinkDB and other systems. AFAIK it's easier to implement and understand than Zookeeper's consensus solution. https://raft.github.io https://raft.github.io
  - deleted 7y ago
    [deleted]
  - ypcx 7y ago
    Kafka was thinking about etcd: https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+... Considering most Kafkas will probably run in Kubernetes at some point, they could have shared the etcd used by Kubernetes.
    - Plugawy 7y ago
      Sounds like a bad idea, no? Last ting you want is whole cluster going down because Kafka is misbehaving
      - ypcx 7y ago
        I don't know. First, there may be some separation achievable - but I'm just guessing. Second, if Kafka is your primary workload, you don't want it to misbehave in any case. But of course I get your point. I'm just saying I've seen people thinking about it that way in some Github Issues.
    - edude03 7y ago
      There is also Zetcd[0] if you want to use etcd and kafka together today. We use it right now and it seems to work [0] https://coreos.com/blog/introducing-zetcd https://coreos.com/blog/introducing-zetcd
- leetbulb 7y ago
  There are other proven solutions that can be embedded such as Raft.
- chrsig 7y ago
  Yes, on both counts. However, requiring zookeeper as a dependency is also a barrier to adoption and increases the operational surface area, so offloading consensus management to zookeeper is not without trade offs.
- binarymax 7y ago
  My thoughts exactly. I don't work with Kafka, but I do heavy work with Solr that also has ZK as a dependency. To end users of whatever platform it is keeping together, Zookeeper is often seen as an unwanted dependency. The unwanted sentiment arises because maintaining zookeeper is not a gimme, and it takes additional knowledge and overhead to maintain. You need a quorum to be kept alive with an odd number (greater than one) of instances. So the question often arises, "how can we get rid of zookeeper?".
- minxomat 7y ago
  Not to mention that ZooKeeper passed Jepsen (on the first try IIRC). Don't replace something so extraordinarily dependable just because it's a dependency.
  - rad_gruchalski 7y ago
    Zookeeper can also be embedded.
    - justnoise 7y ago
      As can etcd. Edit: Oh right, the fact that etcd is golang might make that an issue for Kafka...
- helper 7y ago
  As an operator, having fewer dependencies I have to deploy and worry about is a boon to my productivity. If you were building a kafka like system in house for private use only, then yes I agree with your sentiment. If you are building something to be used by hundreds or thousands of organizations then the cost benefit tradeoffs shift to where it probably makes sense to pull the consensus logic into the primary application itself.
  - cafxx 7y ago
    As an operator, shouldn't you worry if something battle-tested, that implements extremely critical functions that are tricky to implement correctly, is replaced with something that is unproven? As someone who enjoys sleeping at night, I wouldn't poke something like that with a stick before it has matured for a few years, and before many brave (?) operators have smoothed out most rough edges by landing on them repeatedly with their faces.
    - helper 7y ago
      Yeah, don't deploy it when its in beta or its first few releases. At some point it will be stable and safe at which point everyone from then on will benefit.
  - agallego 7y ago
    Note that you still have the same number of fault domains if you read the actual proposal. It does not simplify the system. It is simply replacing the ZK ensemble w/ its own raft impl. Same number of JVMs, same operational complexity.
    - helper 7y ago
      Thanks for pointing that out. Thats what I get for not actually reading the proposal.
- mangatmodi 7y ago
  Finally somebody said it. Yes it is hard. Zookeeper is time tested and I don't think there is anything revolutionary to keep state less than 2n nodes. Either Kafka will compromise on consistency or give a solution which is as performant as Zookeeper. May be they are looking for something which is more specific to Kafka and can be done in a much easier algorithm than Zab?
- czinck 7y ago
  Reading the proposal, it seems like they want to replace it because the ZooKeeper API is causing some issues for them, so even if ZooKeeper has perfect consensus, Kafka itself won't necessarily. From the "Metadata as an Event Log" section > although ZooKeeper is the store of record, the state in ZooKeeper often doesn't match the state that is held in memory in the controller Maybe it would be better to put the effort in to changing ZooKeeper instead of writing their own consensus system in Kafka, but I would assume they know that tradeoff better than us since they work so closely with ZooKeeper.
  - rad_gruchalski 7y ago
    I do find that statement really confusing. That implies they could have errors in the code handling the consensus? Hard to believe but otoh, they’re not really saying anything about zookeeper causing problems. Maybe the correct solution is to understand why the states don’t match in the first place?
- atombender 7y ago
  No, contrary to the sibling comments I would disagree that consensus is difficult. The theory can be somewhat daunting, and ZooKeeper's consensus protocol is complex. The most advanced consensus protocol today, Paxos (and its variations), is notoriously hard to understand. But then we got the Raft paper, which has single-handedly essentially managed to commoditize distributed consensus. Raft is an elegant and simple protocol that has been implemented in many languages. To implement primitives like leader election and consistent, distributed reads/writes, you can grab an off-the-shelf library like Hashicorp's Raft library, which does all the heavy lifting as long as you implement the state storage yourself. It's absurdly trivial to turn a single-node app into a replicated, fully redundant one. Of course, a project like Kafka might have particular requirements (I wouldn't know) that would mean Raft would not be a suitable solution for them, or would require modifications to the protocol. CockroachDB, for example, had to modify Raft ("MultiRaft") in order to achieve the scalability they needed within Raft's paradigm. Raft is a starting point more than a finished solution.
  - necubi 7y ago
    I'd read through some of Aphyr's jepsen tests [0] if you think implementing consensus algorithms correctly is easy. Nearly every distributed database he's looked at has consistency bugs (with the notable exception of zookeeper). [0] https://jepsen.io/analyses https://jepsen.io/analyses
    - atombender 7y ago
      I've read every single one of Aphyr's tests. Aphyr's analyses probe a broader range of distributed consistency problems in replicated stateful databases. Consensus is a necessary mechanism to achieve distributed consistency, but if you look at the problems highlighted in products like CockroachDB and TiDB, they relate to secondary effects of failures to ensure linearizability of reads and writes. You can have a perfectly functioning consensus algorithm and still battle inconsistent reads across nodes, loss of data on node failure or network splits, write skew, and so on. For example, TiKV uses Raft, yet the problems uncovered in TiKV have nothing to do with Raft itself. Conversely, those found in Elasticsearch are absolutely caused by its consensus algorithm, Zen, which was invented in ignorance of modern consensus research (Paxos, etc.). If you read my comment carefully, you'll note that it's above all an endorsement of Raft, which is a milestone in that it democratizes advanced consensus and makes it a tool anyone can use. My point isn't that anyone can trivially whip up an infinitely scalable, consistent database because consensus is a solved problem, but that we have solid foundational theory to build on, to the point where it is a well-understood problem (albeit with many complicated implementations) and not black magic. Also, I did not use the term "easy". But I also think the OP's phrasing "extremely hard" is not true, unless your requirements are extremely hard – which, again, may be the case with Kafka; scalability tends to be diametrically opposed to consistency.
- RantyDave 7y ago
  Exactly, and tbh this kind of thing pisses me off. The "Kafka people" seem to imagine that what Zookeeper does is dead simple and they can bash together their own in a weekend and it'll be better. Zookeeper is great. Weird, for sure, and a bit of a pain in the arse. But it will neither lie to you nor lose data and this is much, much harder to achieve than you might think.
mayank 7y ago
Note: this is not about replacing ZooKeeper in general with Kafka as the title might suggest, it's about Kafka considering an alternative to its internal use of ZooKeeper: "Currently, Kafka uses ZooKeeper to store its metadata about partitions and brokers, and to elect a broker to be the Kafka Controller. We would like to remove this dependency on ZooKeeper. This will enable us to manage metadata in a more scalable and robust way, enabling support for more partitions. It will also simplify the deployment and configuration of Kafka"
- sonabinu 7y ago
  The current heading is ambiguous as well.
  - jimktrains2 7y ago
    Honestly, the new one is even less informative than the original.
    - tus88 7y ago
      Well if the original gave the impression that kafka and zk were the same kind of software it must have been pretty bad.
james-mcelwain 7y ago
There's a toy Kafka implementation written in Go that attempts to do this: https://github.com/travisjeffery/jocko https://github.com/travisjeffery/jocko Previous HN discussion: https://news.ycombinator.com/item?id=13449728 https://news.ycombinator.com/item?id=13449728
- rad_gruchalski 7y ago
  The author now works for Confluent.
kevsim 7y ago
In the days before hosted Kafka implementations were readily available, I tried to set up Kafka in our AWS infrastructure. Getting Zookeeper working in a world based on autoscaling groups was a nightmare. It felt like it was built for the days where each server was a special snowflake. Looking forward to seeing if this gains traction.
- rad_gruchalski 7y ago
  This is just an FYI type of comment. with zookeeper, one needs an odd number of machines, usually not greater than seven. Servers are indexed from 1 to n. They’re replaced one for one, so the server id matches. Scaling it up and down is a bit of a pita because every server needs to know all other servers. Changing the ensemble members requires restarting every zookeeper instance because zookeeper doesn’t support config reload. Zookeeper does expect a single “host” for every server but it runs perfectly fine with docker. But that’s no different from consul or etcd.
  - vetrom 7y ago
    Zookeeper has implemented dynamic reconfiguration since circa 2013-2015ish, but it required support for the feature in clients, as well as servers. That said, implementing an autoscaler that manages that as well was not trivial, in my experience.
    - lars_francke 7y ago
      This is only half true. Yes, the dynamic reconfiguration features has been in trunk for a while but ZooKeeper 3.5 which is the first GA release to include this has only been released 4-8 weeks ago.
  - thomaslee 7y ago
    Another FYI type comment I guess. :) Some of my more gripey statements here may be outdated info, so DYOR I guess. Dynamic reconfig in 3.5 addresses the "restarting every zookeeper instance" problem. [0] You stand up an initial quorum with seed config, then tie in new servers with "reconfig -add". Not sure how well it would tie into cloudy autoscaling stuff though. I wouldn't start there myself. A much bigger pain IMO is the handling of DNS in the official Java ZK client earlier than 3.4.13/3.5.5 (and by association, Curator, ZkClient, etc.). [1] The former was released mid 2018 and the latter this year, so tons of stuff out there that just won't find a host if IPs change. If you "own" all the clients it's maybe not a problem, but if you've got a lot of services owned by a ton of teams it's ... challenging. Even with the fix for ZOOKEEPER-2184 in place I'm pretty sure DNS lookups are only retried if a connect fails, so there's still the issue of IPs "swapping" unexpectedly at the wrong time in cloud environments which can lead to a ZK server in cluster A talking to a ZK server in cluster B (or worse: clients of cluster A talking to cluster B mistakenly thinking that they're talking to cluster A). I'm sure this problem's not unique to ZK though. Authentication helps prevent the worst-case scenarios, but I'm not sure if it helps from an uptime perspective. TL;DR: ZK in the cloud can get messy (even if you play it relatively "safe"). [0] https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.ht... [1] https://issues.apache.org/jira/browse/ZOOKEEPER-2184 https://issues.apache.org/jira/browse/ZOOKEEPER-2184
ccleve 7y ago
I have written a Raft implementation in Java. If anyone from the Kafka project wants it, please contact me. It's not open source, but I own it and could make it so.
- lars_francke 7y ago
  Not to take away from what you did but the Apache Foundation actually has Ratis which is a Raft implementation: http://ratis.incubator.apache.org/ http://ratis.incubator.apache.org/
linuxhansl 7y ago
I don't quite understand why everybody and their mother are trying to remove Zookeeper from their setup. In my past I've seen this many times and each time people went back to Zookeeper after a while, because - as it turns out - consensus is hard; and Zookeeper is battle hardened.
- atombender 7y ago
  First, because it's yet another dependency. Consensus-based systems like CockroachDB, Dgraph, Cassandra, Riak, Elasticsearch, ActorDB, Rqlite, Aerospike, YugaByte, etc. are wonderfully easy to deploy because they can run with no external dependencies. (Their consensus protocols may have different problems, but that's beside the point.) Secondly, it's a pretty heavy dependency. The JVM is RAM-hungry and it's difficult to ensure that it always has enough RAM. Running multiple JVM apps on a single node must be done carefully to make sure each app has enough headroom. It consumes considerably more RAM than Etcd and Consul. Thirdly, I think it's fair to say that ZK is showing its age. It's notorious for being hard to manage (see the other comments in this thread), with a fairly old design (based on the now-ancient Google Chubby paper) that, while resilient, is also less flexible than some other competing designs.
superapc 7y ago
we (alluxio.io) have gone through a similar process by replacing Zookeeper with CopyCat (a raft implementation) for both leader election and storing shared journal since Alluxio 2.0 . Works pretty well
rhacker 7y ago
It would be nice if all the cloud vendors agreed on a key/value and/or consensus protocol that all servers in a cluster can connect to - and maybe even supported via docker, natively even if there's just one cluster member. Like plug-n-play for clustering tech. (Bonjour basically but suitable for cloud/enterprise software)
PunksATawnyFill 7y ago
Whoever wrote this doesn't know WTF a quorum is.
- thilog 7y ago
  Care to elaborate?
noahdesu 7y ago
This sounds really similar to some of the work coming out of Vectorized on Redpanda [0]. They're building a Kafka API compatible system in C++ that's apparently achieving significant performance gains (throughput, tail latency) while maintaining operational simplicity. [0]: https://vectorized.io/redpanda https://vectorized.io/redpanda
- shaklee3 7y ago
  Where do you see performance comparisons to Kafka? As far as I can tell, the product you linked doesn't exist outside of a blog, so this reads like a marketing post.
MrBuddyCasino 7y ago
To all the people wondering why "replace battle tested ZK" and how "consensus is hard": its right there under the motivation header: > enabling support for more partitions I don't know if anyone of you ever ran a high-throughput Kafka cluster with a large number of partitions (as in, thousands of them), but its not pretty. Rebalancing can easily take half an hour after a rollout, and throughput is degraded during that time. We recently had to move to shared topics because it became untenable. This is a very welcome change!
simtel20 7y ago
Is no-one else running into FastLeaderElectionFailed? When you have a system that writes a lot of offset/transaction info to zookeeper you can push the zxid 32-bit counter to rollover in a matter of days. When this happens it can bring zookeeper to a grinding halt for 15 minutes after 2 nodes try to nominate themselves for leadership and the rest of the cluster sits back and waits for a timeout. https://issues.apache.org/jira/browse/ZOOKEEPER-2164 https://issues.apache.org/jira/browse/ZOOKEEPER-2164 https://issues.apache.org/jira/browse/ZOOKEEPER-2791 https://issues.apache.org/jira/browse/ZOOKEEPER-2791 Requests (can't find them in JIRA at the moment, so I need to paraphrase) in the past to have a call to initiate a controlled leadership move to another node have been turned down as "you don't need this" yet leadership election fails in some circumstances! In addition there's no command or configuration to disable FastLeaderElection. So the zookeeper maintainers keep operators limited to having to flip nodes off and on again, which is really a bad way to manage software because it impacts clients as well as leadership (and even if clients recover, most code that I've seen like to make some noise when zk connections flap). I would really like to eliminate all use cases for zookeeper where there is a chance that the zxid will exceed the size of its 32-bit counter component in the span of, say, a decade so that as an operator I don't have to set alerts on the zxid counter creeping up, and having to reset zookeeper and restart all of its clients (many versions of many zookeeper clients don't retry after connection loss, don't retry after a timeout, don't cope with the primary connection failing, will have totally given up after 15 minutes, etc.). I think that the kafka maintainers have been doing a better job of actively maintaining their code and ensuring it works in adverse conditions, so I'm on board with this proposal. Zookeeper isn't magic, it's just pretty good at most of what it does, and I think that projects that understand when they've pushed zookeeper into a bad corner may benefit from this kind of move, if they also have a good idea of how they can do better.
camiloaguilar 7y ago
I’m not too convinced of the approach. I’ve been anxiously waiting for https://vectorized.io/ https://vectorized.io/ to release their message queue. It is built in modern C++, uses ScyllaDB Seastar framework to do IO scheduling in userspace, with better mechanical sympathy. And like Hashicorp’s Nomad and Vault, which I’m a fan of, it has built-in distributed consensus and easy operation.

139 points by telekid 7y ago

Isn't managing consensus extremely hard to do? Wouldn't one want to rely on a proven solution rather than spinning up a new solution?

empath75 7y ago
Well, there's etcd, consul, etc.. edit: i guess i should RTFA -- looks like they are going to build raft into kafka -- at this point, I'd think raft libraries are fairly mature.
james-mcelwain 7y ago
There are a number of Java libraries that implement Raft already -- it's a fairly well understood algorithm.
- rad_gruchalski 7y ago
  TIL: there is an Apache Incubator raft project: http://ratis.incubator.apache.org/ http://ratis.incubator.apache.org/.
Plugawy 7y ago
The document mentions using Raft for consensus and coordination - it's the same approach used by Etcd, Consul, Serf, RethinkDB and other systems. AFAIK it's easier to implement and understand than Zookeeper's consensus solution. https://raft.github.io https://raft.github.io
- deleted 7y ago
  [deleted]
- ypcx 7y ago
  Kafka was thinking about etcd: https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+... Considering most Kafkas will probably run in Kubernetes at some point, they could have shared the etcd used by Kubernetes.
  - Plugawy 7y ago
    Sounds like a bad idea, no? Last ting you want is whole cluster going down because Kafka is misbehaving
    - ypcx 7y ago
      I don't know. First, there may be some separation achievable - but I'm just guessing. Second, if Kafka is your primary workload, you don't want it to misbehave in any case. But of course I get your point. I'm just saying I've seen people thinking about it that way in some Github Issues.
  - edude03 7y ago
    There is also Zetcd[0] if you want to use etcd and kafka together today. We use it right now and it seems to work [0] https://coreos.com/blog/introducing-zetcd https://coreos.com/blog/introducing-zetcd
leetbulb 7y ago
There are other proven solutions that can be embedded such as Raft.
chrsig 7y ago
Yes, on both counts. However, requiring zookeeper as a dependency is also a barrier to adoption and increases the operational surface area, so offloading consensus management to zookeeper is not without trade offs.
binarymax 7y ago
My thoughts exactly. I don't work with Kafka, but I do heavy work with Solr that also has ZK as a dependency. To end users of whatever platform it is keeping together, Zookeeper is often seen as an unwanted dependency. The unwanted sentiment arises because maintaining zookeeper is not a gimme, and it takes additional knowledge and overhead to maintain. You need a quorum to be kept alive with an odd number (greater than one) of instances. So the question often arises, "how can we get rid of zookeeper?".
minxomat 7y ago
Not to mention that ZooKeeper passed Jepsen (on the first try IIRC). Don't replace something so extraordinarily dependable just because it's a dependency.
- rad_gruchalski 7y ago
  Zookeeper can also be embedded.
  - justnoise 7y ago
    As can etcd. Edit: Oh right, the fact that etcd is golang might make that an issue for Kafka...
helper 7y ago
As an operator, having fewer dependencies I have to deploy and worry about is a boon to my productivity. If you were building a kafka like system in house for private use only, then yes I agree with your sentiment. If you are building something to be used by hundreds or thousands of organizations then the cost benefit tradeoffs shift to where it probably makes sense to pull the consensus logic into the primary application itself.
- cafxx 7y ago
  As an operator, shouldn't you worry if something battle-tested, that implements extremely critical functions that are tricky to implement correctly, is replaced with something that is unproven? As someone who enjoys sleeping at night, I wouldn't poke something like that with a stick before it has matured for a few years, and before many brave (?) operators have smoothed out most rough edges by landing on them repeatedly with their faces.
  - helper 7y ago
    Yeah, don't deploy it when its in beta or its first few releases. At some point it will be stable and safe at which point everyone from then on will benefit.
- agallego 7y ago
  Note that you still have the same number of fault domains if you read the actual proposal. It does not simplify the system. It is simply replacing the ZK ensemble w/ its own raft impl. Same number of JVMs, same operational complexity.
  - helper 7y ago
    Thanks for pointing that out. Thats what I get for not actually reading the proposal.
mangatmodi 7y ago
Finally somebody said it. Yes it is hard. Zookeeper is time tested and I don't think there is anything revolutionary to keep state less than 2n nodes. Either Kafka will compromise on consistency or give a solution which is as performant as Zookeeper. May be they are looking for something which is more specific to Kafka and can be done in a much easier algorithm than Zab?
czinck 7y ago
Reading the proposal, it seems like they want to replace it because the ZooKeeper API is causing some issues for them, so even if ZooKeeper has perfect consensus, Kafka itself won't necessarily. From the "Metadata as an Event Log" section > although ZooKeeper is the store of record, the state in ZooKeeper often doesn't match the state that is held in memory in the controller Maybe it would be better to put the effort in to changing ZooKeeper instead of writing their own consensus system in Kafka, but I would assume they know that tradeoff better than us since they work so closely with ZooKeeper.
- rad_gruchalski 7y ago
  I do find that statement really confusing. That implies they could have errors in the code handling the consensus? Hard to believe but otoh, they’re not really saying anything about zookeeper causing problems. Maybe the correct solution is to understand why the states don’t match in the first place?
atombender 7y ago
No, contrary to the sibling comments I would disagree that consensus is difficult. The theory can be somewhat daunting, and ZooKeeper's consensus protocol is complex. The most advanced consensus protocol today, Paxos (and its variations), is notoriously hard to understand. But then we got the Raft paper, which has single-handedly essentially managed to commoditize distributed consensus. Raft is an elegant and simple protocol that has been implemented in many languages. To implement primitives like leader election and consistent, distributed reads/writes, you can grab an off-the-shelf library like Hashicorp's Raft library, which does all the heavy lifting as long as you implement the state storage yourself. It's absurdly trivial to turn a single-node app into a replicated, fully redundant one. Of course, a project like Kafka might have particular requirements (I wouldn't know) that would mean Raft would not be a suitable solution for them, or would require modifications to the protocol. CockroachDB, for example, had to modify Raft ("MultiRaft") in order to achieve the scalability they needed within Raft's paradigm. Raft is a starting point more than a finished solution.
- necubi 7y ago
  I'd read through some of Aphyr's jepsen tests [0] if you think implementing consensus algorithms correctly is easy. Nearly every distributed database he's looked at has consistency bugs (with the notable exception of zookeeper). [0] https://jepsen.io/analyses https://jepsen.io/analyses
  - atombender 7y ago
    I've read every single one of Aphyr's tests. Aphyr's analyses probe a broader range of distributed consistency problems in replicated stateful databases. Consensus is a necessary mechanism to achieve distributed consistency, but if you look at the problems highlighted in products like CockroachDB and TiDB, they relate to secondary effects of failures to ensure linearizability of reads and writes. You can have a perfectly functioning consensus algorithm and still battle inconsistent reads across nodes, loss of data on node failure or network splits, write skew, and so on. For example, TiKV uses Raft, yet the problems uncovered in TiKV have nothing to do with Raft itself. Conversely, those found in Elasticsearch are absolutely caused by its consensus algorithm, Zen, which was invented in ignorance of modern consensus research (Paxos, etc.). If you read my comment carefully, you'll note that it's above all an endorsement of Raft, which is a milestone in that it democratizes advanced consensus and makes it a tool anyone can use. My point isn't that anyone can trivially whip up an infinitely scalable, consistent database because consensus is a solved problem, but that we have solid foundational theory to build on, to the point where it is a well-understood problem (albeit with many complicated implementations) and not black magic. Also, I did not use the term "easy". But I also think the OP's phrasing "extremely hard" is not true, unless your requirements are extremely hard – which, again, may be the case with Kafka; scalability tends to be diametrically opposed to consistency.
RantyDave 7y ago
Exactly, and tbh this kind of thing pisses me off. The "Kafka people" seem to imagine that what Zookeeper does is dead simple and they can bash together their own in a weekend and it'll be better. Zookeeper is great. Weird, for sure, and a bit of a pain in the arse. But it will neither lie to you nor lose data and this is much, much harder to achieve than you might think.

mayank 7y ago

Note: this is not about replacing ZooKeeper in general with Kafka as the title might suggest, it's about Kafka considering an alternative to its internal use of ZooKeeper: "Currently, Kafka uses ZooKeeper to store its metadata about partitions and brokers, and to elect a broker to be the Kafka Controller. We would like to remove this dependency on ZooKeeper. This will enable us to manage metadata in a more scalable and robust way, enabling support for more partitions. It will also simplify the deployment and configuration of Kafka"

sonabinu 7y ago
The current heading is ambiguous as well.
- jimktrains2 7y ago
  Honestly, the new one is even less informative than the original.
  - tus88 7y ago
    Well if the original gave the impression that kafka and zk were the same kind of software it must have been pretty bad.

james-mcelwain 7y ago

There's a toy Kafka implementation written in Go that attempts to do this: https://github.com/travisjeffery/jocko https://github.com/travisjeffery/jocko Previous HN discussion: https://news.ycombinator.com/item?id=13449728 https://news.ycombinator.com/item?id=13449728

rad_gruchalski 7y ago
The author now works for Confluent.

kevsim 7y ago

In the days before hosted Kafka implementations were readily available, I tried to set up Kafka in our AWS infrastructure. Getting Zookeeper working in a world based on autoscaling groups was a nightmare. It felt like it was built for the days where each server was a special snowflake. Looking forward to seeing if this gains traction.

rad_gruchalski 7y ago
This is just an FYI type of comment. with zookeeper, one needs an odd number of machines, usually not greater than seven. Servers are indexed from 1 to n. They’re replaced one for one, so the server id matches. Scaling it up and down is a bit of a pita because every server needs to know all other servers. Changing the ensemble members requires restarting every zookeeper instance because zookeeper doesn’t support config reload. Zookeeper does expect a single “host” for every server but it runs perfectly fine with docker. But that’s no different from consul or etcd.
- vetrom 7y ago
  Zookeeper has implemented dynamic reconfiguration since circa 2013-2015ish, but it required support for the feature in clients, as well as servers. That said, implementing an autoscaler that manages that as well was not trivial, in my experience.
  - lars_francke 7y ago
    This is only half true. Yes, the dynamic reconfiguration features has been in trunk for a while but ZooKeeper 3.5 which is the first GA release to include this has only been released 4-8 weeks ago.
- thomaslee 7y ago
  Another FYI type comment I guess. :) Some of my more gripey statements here may be outdated info, so DYOR I guess. Dynamic reconfig in 3.5 addresses the "restarting every zookeeper instance" problem. [0] You stand up an initial quorum with seed config, then tie in new servers with "reconfig -add". Not sure how well it would tie into cloudy autoscaling stuff though. I wouldn't start there myself. A much bigger pain IMO is the handling of DNS in the official Java ZK client earlier than 3.4.13/3.5.5 (and by association, Curator, ZkClient, etc.). [1] The former was released mid 2018 and the latter this year, so tons of stuff out there that just won't find a host if IPs change. If you "own" all the clients it's maybe not a problem, but if you've got a lot of services owned by a ton of teams it's ... challenging. Even with the fix for ZOOKEEPER-2184 in place I'm pretty sure DNS lookups are only retried if a connect fails, so there's still the issue of IPs "swapping" unexpectedly at the wrong time in cloud environments which can lead to a ZK server in cluster A talking to a ZK server in cluster B (or worse: clients of cluster A talking to cluster B mistakenly thinking that they're talking to cluster A). I'm sure this problem's not unique to ZK though. Authentication helps prevent the worst-case scenarios, but I'm not sure if it helps from an uptime perspective. TL;DR: ZK in the cloud can get messy (even if you play it relatively "safe"). [0] https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.ht... [1] https://issues.apache.org/jira/browse/ZOOKEEPER-2184 https://issues.apache.org/jira/browse/ZOOKEEPER-2184

ccleve 7y ago

I have written a Raft implementation in Java. If anyone from the Kafka project wants it, please contact me. It's not open source, but I own it and could make it so.

lars_francke 7y ago
Not to take away from what you did but the Apache Foundation actually has Ratis which is a Raft implementation: http://ratis.incubator.apache.org/ http://ratis.incubator.apache.org/

linuxhansl 7y ago

I don't quite understand why everybody and their mother are trying to remove Zookeeper from their setup. In my past I've seen this many times and each time people went back to Zookeeper after a while, because - as it turns out - consensus is hard; and Zookeeper is battle hardened.

atombender 7y ago
First, because it's yet another dependency. Consensus-based systems like CockroachDB, Dgraph, Cassandra, Riak, Elasticsearch, ActorDB, Rqlite, Aerospike, YugaByte, etc. are wonderfully easy to deploy because they can run with no external dependencies. (Their consensus protocols may have different problems, but that's beside the point.) Secondly, it's a pretty heavy dependency. The JVM is RAM-hungry and it's difficult to ensure that it always has enough RAM. Running multiple JVM apps on a single node must be done carefully to make sure each app has enough headroom. It consumes considerably more RAM than Etcd and Consul. Thirdly, I think it's fair to say that ZK is showing its age. It's notorious for being hard to manage (see the other comments in this thread), with a fairly old design (based on the now-ancient Google Chubby paper) that, while resilient, is also less flexible than some other competing designs.

superapc 7y ago

we (alluxio.io) have gone through a similar process by replacing Zookeeper with CopyCat (a raft implementation) for both leader election and storing shared journal since Alluxio 2.0 . Works pretty well

rhacker 7y ago

It would be nice if all the cloud vendors agreed on a key/value and/or consensus protocol that all servers in a cluster can connect to - and maybe even supported via docker, natively even if there's just one cluster member. Like plug-n-play for clustering tech. (Bonjour basically but suitable for cloud/enterprise software)

PunksATawnyFill 7y ago

Whoever wrote this doesn't know WTF a quorum is.

thilog 7y ago
Care to elaborate?

noahdesu 7y ago

This sounds really similar to some of the work coming out of Vectorized on Redpanda [0]. They're building a Kafka API compatible system in C++ that's apparently achieving significant performance gains (throughput, tail latency) while maintaining operational simplicity. [0]: https://vectorized.io/redpanda https://vectorized.io/redpanda

shaklee3 7y ago
Where do you see performance comparisons to Kafka? As far as I can tell, the product you linked doesn't exist outside of a blog, so this reads like a marketing post.

MrBuddyCasino 7y ago

To all the people wondering why "replace battle tested ZK" and how "consensus is hard": its right there under the motivation header: > enabling support for more partitions I don't know if anyone of you ever ran a high-throughput Kafka cluster with a large number of partitions (as in, thousands of them), but its not pretty. Rebalancing can easily take half an hour after a rollout, and throughput is degraded during that time. We recently had to move to shared topics because it became untenable. This is a very welcome change!

simtel20 7y ago

Is no-one else running into FastLeaderElectionFailed? When you have a system that writes a lot of offset/transaction info to zookeeper you can push the zxid 32-bit counter to rollover in a matter of days. When this happens it can bring zookeeper to a grinding halt for 15 minutes after 2 nodes try to nominate themselves for leadership and the rest of the cluster sits back and waits for a timeout. https://issues.apache.org/jira/browse/ZOOKEEPER-2164 https://issues.apache.org/jira/browse/ZOOKEEPER-2164 https://issues.apache.org/jira/browse/ZOOKEEPER-2791 https://issues.apache.org/jira/browse/ZOOKEEPER-2791 Requests (can't find them in JIRA at the moment, so I need to paraphrase) in the past to have a call to initiate a controlled leadership move to another node have been turned down as "you don't need this" yet leadership election fails in some circumstances! In addition there's no command or configuration to disable FastLeaderElection. So the zookeeper maintainers keep operators limited to having to flip nodes off and on again, which is really a bad way to manage software because it impacts clients as well as leadership (and even if clients recover, most code that I've seen like to make some noise when zk connections flap). I would really like to eliminate all use cases for zookeeper where there is a chance that the zxid will exceed the size of its 32-bit counter component in the span of, say, a decade so that as an operator I don't have to set alerts on the zxid counter creeping up, and having to reset zookeeper and restart all of its clients (many versions of many zookeeper clients don't retry after connection loss, don't retry after a timeout, don't cope with the primary connection failing, will have totally given up after 15 minutes, etc.). I think that the kafka maintainers have been doing a better job of actively maintaining their code and ensuring it works in adverse conditions, so I'm on board with this proposal. Zookeeper isn't magic, it's just pretty good at most of what it does, and I think that projects that understand when they've pushed zookeeper into a bad corner may benefit from this kind of move, if they also have a good idea of how they can do better.

camiloaguilar 7y ago

I’m not too convinced of the approach. I’ve been anxiously waiting for https://vectorized.io/ https://vectorized.io/ to release their message queue. It is built in modern C++, uses ScyllaDB Seastar framework to do IO scheduling in userspace, with better mechanical sympathy. And like Hashicorp’s Nomad and Vault, which I’m a fan of, it has built-in distributed consensus and easy operation.