16 ms·

Kafka Removing Zookeeper Dependency

314 points by manigandham 6y ago

fourseventy 6y ago
This is cool. One thing I love about Cassandra is that you don't need Zookeeper, its just a bunch of homogeneous nodes.
- mavelikara 6y ago
  Same with Elastic Search.
ironfootnz 6y ago
Well... it’s just Kafssandra now.
hugofirth 6y ago
I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but it also means that more people are familiar with the quirks, limitations, requirements etc.... of these tools. For example, I think I would be frustrated to hear that K8s were implementing their own raft.
- smarterclayton 6y ago
  The difference is that Kubernetes isn’t a data store - we’d have to implement the full functionality of etcd. Kafka is a full featured data store that doesn’t need 80% of what zookeeper does. This was actually an early complaint leveled against Kubernetes for things like proxying at the node level or implementing DNS. “Don’t reinvent the wheel!” Sometimes better administrative experiences exist only after a component absorbs some function previous systems expose. Sometimes it’s better to own the parts of the problem that make your system simpler.
  - justicezyx 6y ago
    No etcd don't need to be replicated for k8s API server. Basically versioning + some form of fault tolerance is sufficient for k8s API server.
- sagichmal 6y ago
  Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.
  - saurik 6y ago
    I feel like this just argues that ZooKeeper should be a library that can be embedded in the servers of other projects (even if it is conceptually a separate server in the same memory space listening on its own ports; but like, it is otherwise entirely hidden inside of the other program and is entirely managed and configured by it also).
    - chucky_z 6y ago
      Isn't this exactly what lead to Hashicorp's memberlist library being used all over in random projects?
    - skyde 6y ago
      100 % agree we need good (etch/zookeeper) as a library. I wrote something similar but that provide only (lock/lease) using paxos. The problem with raft ETCD and zookeeper replicates state machine design is that : - machine leaving and joining the ensemble dynamically is something hard to do correctly - optimal quorum size is no more than 5, you can setup other node as observer but it’s hard to decide which of 1000 node should be in the quorum ...
      - lima 6y ago
        The etcd server can be embedded in other projects, and their Raft library is used by plenty of other projects like CockroachDB.
        skyde 6y ago
        yes it can but it’s a real pain compared to hashicorp version. I really hope they clean up the library to have a simple API. This way it could become the standard for GO and we can stop wasting effort on multiple raft implementation.
- gregwebs 6y ago
  There are a lot of shared raft libraries now. It is much easier to share library code than a piece of a distributed system.
- manigandham 6y ago
  This is the difference between a library and a framework/project. For example, there's a great Raft library for Go [1] that any project can use to implement distributed consensus without a separate running program. I find this to be a better approach with the same collective development and community testing advantages but without requiring more operational overhead. 1. https://github.com/hashicorp/raft https://github.com/hashicorp/raft
  - skyde 6y ago
    this lib u is great and 1000 time cleaner better tested and easier to use compared to trying to reuse etcd raft implementation :-)
- whorleater 6y ago
  If ZK wasn't such a pain in the ass to manage I'd agree with you.
  - skyde 6y ago
    this pain come from paxos raft or Zab algorithm used by concensus system. I don’t think it can be avoided.
- kirstenbirgit 6y ago
  I generally think it makes sense to start with readily available tools, but as a project grows and becomes more specialized, it often shows that the tool doesn't quite fit, doesn't quite have the same performance characteristics, or otherwise not a good fit anymore. And then it's time for something better. Or, it works out, and you don't switch it out (e.g. etcd in k8s)
eatonphil 6y ago
Here's the proposal in full. https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...
nealabq 6y ago
I expect Pulsar to stay with Zookeeper. Kafka currently stores topic and partition info on ZK ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data... ), which can get to be a lot of data. But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable.
- manigandham 6y ago
  True, Pulsar's architecture is more reliable and scalable than Kafka, but it would still be nice to remove any extra dependencies.
- biggestlou 6y ago
  That is correct: Pulsar uses ZooKeeper relatively lightly and only for things strictly necessary for the cluster to run. Everything else (message data, topic info, cursors, schemas) is stored in BookKeeper. Would still be great to see ZooKeeper made superfluous, though.
- colin_mccabe 6y ago
  But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable. Unfortunately, this is not correct. BookKeeper stores a lot of information in ZookKeeper. By extension, Pulsar (which is based on BookKeeper) also stores a lot of metadata there as well. For example, from the BK documentation ( https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.html https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.h... ): An application first creates a ledger before writing to bookies through a local BookKeeper client instance. Upon creating a ledger, a BookKeeper client writes metadata about the ledger to ZooKeeper. Each ledger currently has a single writer. This writer has to execute a close ledger operation before any other client can read from it. If the writer of a ledger does not close a ledger properly because, for example, it has crashed before having the opportunity of closing the ledger, then the next client that tries to open a ledger executes a procedure to recover it. As closing a ledger consists essentially of writing the last entry written to a ledger to ZooKeeper, the recovery procedure simply finds the last entry written correctly and writes it to ZooKeeper.
tbrock 6y ago
Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper dep, JVM foundation)
- closeparen 6y ago
  Keystores are clunky, but other than that, why exactly?
  - tbrock 6y ago
    Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?
    - staticassertion 6y ago
      But Go doesn't "figure it out" for you. It decides for you - there's no "figure it out" step. That's the power of having tunables.
      - tbrock 6y ago
        Then they chose better defaults because I never have to think about it.
        growse 6y ago
        Just because the defaults are better for you doesn't mean they're better for everyone.
    - nhoughto 6y ago
      Sometimes having more control over GC is useful tho, could save a re-write. https://blog.discord.com/why-discord-is-switching-from-go-to-rust-a190bbca2b1f https://blog.discord.com/why-discord-is-switching-from-go-to...
kungfufrog 6y ago
When will this be actually released and available for use? I'd love to ditch Zookeeper (the weak part that falls over more regularly) in our current Kafka cluster.
xorchasiv 6y ago
I might be out of my realm here but would it be more efficient for the Kafka brokers to use a gossip protocol to distribute the metadata?
- jdean677 6y ago
  Gossip protocol is useful for propagating node failures or events that are fine with being eventually consistent. It cannot be used for leader election as these events are time sensitive and needs to be consistent across the cluster within a short duration, that is the reason we have raft & paxos.
  - xorchasiv 6y ago
    Ah I see, thank you for the clarification
lchengify 6y ago
I couldn't be more excited about this! Having finished a Kafka project about a year ago, we had so many Zookeeper production and test environment issues that it was the running joke to check Zookeeper first if anything went wrong. Honestly Zookeeper in theory is a great idea: Having a centralized service for maintaining config info saves a lot of heartache when dealing with an open source distributed systems project. But in practice, I've never had a smooth experience getting Zookeeper to run consistently, especially with Kafka. For sure, part of it is that we treat it like oxygen, in that if it's gone for a few seconds everything just dies. But having dealt with similar systems both proprietary and open source, my opinion is that Zookeeper just hasn't risen to the challenges of its users in the past 5 years. If the next generation of software architects want to use open source streaming or distributed systems, Zookeeper needs to be rewritten or removed. Also shout out to the confluent.io team: I never paid for your enterprise license, but without your blog posts, docker images, or slack room, I never would have been able to get Kafka working. Thanks again!
- skyde 6y ago
  i am curious to know why people expect a raft library to be more reliable if embedded inside the kafka controller versus running inside a service like Etcd. in the end broker will do RPC to a service (kafka controller/ etcd) and this service will use raft to replicate the state. It should be exactly the same. And if anything knowing which node are running the raft algorithm help you be more careful with rolling restart and upgrade.
  - gen220 6y ago
    It might sound kind of trite, but it has less to do with the replication algorithm and more to do with the fact that Zookeeper’s source is quite complicated vs, say etcd, so there are more opportunities for subtle bugs to appear. I encourage you to look at the bug section of the change log for zookeeper, and also the feature list of ZK vs etcd. Also, etcd powers many critical open source projects, so there are many institutional eyes that actively contribute to its improvement. IME if we ever encountered an issue at work with ZK, we found it impossible to trace it down to a bug that we could fix and upstream. Etcd’s been easier in this regard.
devin 6y ago
Amazing that people chose to operationalize Kafka with this completely unnecessary dependency. Good riddance!
KenanSulayman 6y ago
Finally! Having had to deal with properly setting zookeeper up on multiple mesos clusters was so painful and recovering from outages via s3 backups so excruciating.. I feel somewhat ashamed that in two out of five cases we simply went with the managed Kafka by AWS because the team didn’t feel confident enough to maintain the cluster.
- gen220 6y ago
  Honestly this is probably the right thing to do. My cynical conspiracy is that the ZK dep was organized by cloud vendors to keep customers coming :) It’s been at least a year and a half since we’ve had severe data loss on our homebrew Kafka cluster, but after the first couple you never look at Kafka+Zk the same way... both iirc were due to leadership election bugs that had been reported many months ago with no progress on a solution. I have no idea how AWS internally puts up with this. I wouldn’t be surprised if they’d replaced the ZK dependency internally years ago.

314 points by manigandham 6y ago

fourseventy 6y ago

This is cool. One thing I love about Cassandra is that you don't need Zookeeper, its just a bunch of homogeneous nodes.

mavelikara 6y ago
Same with Elastic Search.

ironfootnz 6y ago

Well... it’s just Kafssandra now.

hugofirth 6y ago

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but it also means that more people are familiar with the quirks, limitations, requirements etc.... of these tools. For example, I think I would be frustrated to hear that K8s were implementing their own raft.

smarterclayton 6y ago
The difference is that Kubernetes isn’t a data store - we’d have to implement the full functionality of etcd. Kafka is a full featured data store that doesn’t need 80% of what zookeeper does. This was actually an early complaint leveled against Kubernetes for things like proxying at the node level or implementing DNS. “Don’t reinvent the wheel!” Sometimes better administrative experiences exist only after a component absorbs some function previous systems expose. Sometimes it’s better to own the parts of the problem that make your system simpler.
- justicezyx 6y ago
  No etcd don't need to be replicated for k8s API server. Basically versioning + some form of fault tolerance is sufficient for k8s API server.
sagichmal 6y ago
Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.
- saurik 6y ago
  I feel like this just argues that ZooKeeper should be a library that can be embedded in the servers of other projects (even if it is conceptually a separate server in the same memory space listening on its own ports; but like, it is otherwise entirely hidden inside of the other program and is entirely managed and configured by it also).
  - chucky_z 6y ago
    Isn't this exactly what lead to Hashicorp's memberlist library being used all over in random projects?
  - skyde 6y ago
    100 % agree we need good (etch/zookeeper) as a library. I wrote something similar but that provide only (lock/lease) using paxos. The problem with raft ETCD and zookeeper replicates state machine design is that : - machine leaving and joining the ensemble dynamically is something hard to do correctly - optimal quorum size is no more than 5, you can setup other node as observer but it’s hard to decide which of 1000 node should be in the quorum ...
    - lima 6y ago
      The etcd server can be embedded in other projects, and their Raft library is used by plenty of other projects like CockroachDB.
      - skyde 6y ago
        yes it can but it’s a real pain compared to hashicorp version. I really hope they clean up the library to have a simple API. This way it could become the standard for GO and we can stop wasting effort on multiple raft implementation.
gregwebs 6y ago
There are a lot of shared raft libraries now. It is much easier to share library code than a piece of a distributed system.
manigandham 6y ago
This is the difference between a library and a framework/project. For example, there's a great Raft library for Go [1] that any project can use to implement distributed consensus without a separate running program. I find this to be a better approach with the same collective development and community testing advantages but without requiring more operational overhead. 1. https://github.com/hashicorp/raft https://github.com/hashicorp/raft
- skyde 6y ago
  this lib u is great and 1000 time cleaner better tested and easier to use compared to trying to reuse etcd raft implementation :-)
whorleater 6y ago
If ZK wasn't such a pain in the ass to manage I'd agree with you.
- skyde 6y ago
  this pain come from paxos raft or Zab algorithm used by concensus system. I don’t think it can be avoided.
kirstenbirgit 6y ago
I generally think it makes sense to start with readily available tools, but as a project grows and becomes more specialized, it often shows that the tool doesn't quite fit, doesn't quite have the same performance characteristics, or otherwise not a good fit anymore. And then it's time for something better. Or, it works out, and you don't switch it out (e.g. etcd in k8s)

eatonphil 6y ago

Here's the proposal in full. https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...

nealabq 6y ago

I expect Pulsar to stay with Zookeeper. Kafka currently stores topic and partition info on ZK ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data... ), which can get to be a lot of data. But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable.

manigandham 6y ago
True, Pulsar's architecture is more reliable and scalable than Kafka, but it would still be nice to remove any extra dependencies.
biggestlou 6y ago
That is correct: Pulsar uses ZooKeeper relatively lightly and only for things strictly necessary for the cluster to run. Everything else (message data, topic info, cursors, schemas) is stored in BookKeeper. Would still be great to see ZooKeeper made superfluous, though.
colin_mccabe 6y ago
But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable. Unfortunately, this is not correct. BookKeeper stores a lot of information in ZookKeeper. By extension, Pulsar (which is based on BookKeeper) also stores a lot of metadata there as well. For example, from the BK documentation ( https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.html https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.h... ): An application first creates a ledger before writing to bookies through a local BookKeeper client instance. Upon creating a ledger, a BookKeeper client writes metadata about the ledger to ZooKeeper. Each ledger currently has a single writer. This writer has to execute a close ledger operation before any other client can read from it. If the writer of a ledger does not close a ledger properly because, for example, it has crashed before having the opportunity of closing the ledger, then the next client that tries to open a ledger executes a procedure to recover it. As closing a ledger consists essentially of writing the last entry written to a ledger to ZooKeeper, the recovery procedure simply finds the last entry written correctly and writes it to ZooKeeper.

tbrock 6y ago

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper dep, JVM foundation)

closeparen 6y ago
Keystores are clunky, but other than that, why exactly?
- tbrock 6y ago
  Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?
  - staticassertion 6y ago
    But Go doesn't "figure it out" for you. It decides for you - there's no "figure it out" step. That's the power of having tunables.
    - tbrock 6y ago
      Then they chose better defaults because I never have to think about it.
      - growse 6y ago
        Just because the defaults are better for you doesn't mean they're better for everyone.
  - nhoughto 6y ago
    Sometimes having more control over GC is useful tho, could save a re-write. https://blog.discord.com/why-discord-is-switching-from-go-to-rust-a190bbca2b1f https://blog.discord.com/why-discord-is-switching-from-go-to...

kungfufrog 6y ago

When will this be actually released and available for use? I'd love to ditch Zookeeper (the weak part that falls over more regularly) in our current Kafka cluster.

xorchasiv 6y ago

I might be out of my realm here but would it be more efficient for the Kafka brokers to use a gossip protocol to distribute the metadata?

jdean677 6y ago
Gossip protocol is useful for propagating node failures or events that are fine with being eventually consistent. It cannot be used for leader election as these events are time sensitive and needs to be consistent across the cluster within a short duration, that is the reason we have raft & paxos.
- xorchasiv 6y ago
  Ah I see, thank you for the clarification

lchengify 6y ago

I couldn't be more excited about this! Having finished a Kafka project about a year ago, we had so many Zookeeper production and test environment issues that it was the running joke to check Zookeeper first if anything went wrong. Honestly Zookeeper in theory is a great idea: Having a centralized service for maintaining config info saves a lot of heartache when dealing with an open source distributed systems project. But in practice, I've never had a smooth experience getting Zookeeper to run consistently, especially with Kafka. For sure, part of it is that we treat it like oxygen, in that if it's gone for a few seconds everything just dies. But having dealt with similar systems both proprietary and open source, my opinion is that Zookeeper just hasn't risen to the challenges of its users in the past 5 years. If the next generation of software architects want to use open source streaming or distributed systems, Zookeeper needs to be rewritten or removed. Also shout out to the confluent.io team: I never paid for your enterprise license, but without your blog posts, docker images, or slack room, I never would have been able to get Kafka working. Thanks again!

skyde 6y ago
i am curious to know why people expect a raft library to be more reliable if embedded inside the kafka controller versus running inside a service like Etcd. in the end broker will do RPC to a service (kafka controller/ etcd) and this service will use raft to replicate the state. It should be exactly the same. And if anything knowing which node are running the raft algorithm help you be more careful with rolling restart and upgrade.
- gen220 6y ago
  It might sound kind of trite, but it has less to do with the replication algorithm and more to do with the fact that Zookeeper’s source is quite complicated vs, say etcd, so there are more opportunities for subtle bugs to appear. I encourage you to look at the bug section of the change log for zookeeper, and also the feature list of ZK vs etcd. Also, etcd powers many critical open source projects, so there are many institutional eyes that actively contribute to its improvement. IME if we ever encountered an issue at work with ZK, we found it impossible to trace it down to a bug that we could fix and upstream. Etcd’s been easier in this regard.

devin 6y ago

Amazing that people chose to operationalize Kafka with this completely unnecessary dependency. Good riddance!

KenanSulayman 6y ago

Finally! Having had to deal with properly setting zookeeper up on multiple mesos clusters was so painful and recovering from outages via s3 backups so excruciating.. I feel somewhat ashamed that in two out of five cases we simply went with the managed Kafka by AWS because the team didn’t feel confident enough to maintain the cluster.

gen220 6y ago
Honestly this is probably the right thing to do. My cynical conspiracy is that the ZK dep was organized by cloud vendors to keep customers coming :) It’s been at least a year and a half since we’ve had severe data loss on our homebrew Kafka cluster, but after the first couple you never look at Kafka+Zk the same way... both iirc were due to leadership election bugs that had been reported many months ago with no progress on a solution. I have no idea how AWS internally puts up with this. I wouldn’t be surprised if they’d replaced the ZK dependency internally years ago.