4 ms·
i am curious to know why people expect a raft library to be more reliable if embedded inside the kafka controller versus running inside a service like Etcd. in
by skyde 6y ago
i am curious to know why people expect a raft library to be more reliable if embedded inside the kafka controller versus running inside a service like Etcd.
in the end broker will do RPC to a service (kafka controller/ etcd) and this service will use raft to replicate the state.
It should be exactly the same. And if anything knowing which node are running the raft algorithm help you be more careful with rolling restart and upgrade.
- gen220 6y agoIt might sound kind of trite, but it has less to do with the replication algorithm and more to do with the fact that Zookeeper’s source is quite complicated vs, say etcd, so there are more opportunities for subtle bugs to appear. I encourage you to look at the bug section of the change log for zookeeper, and also the feature list of ZK vs etcd. Also, etcd powers many critical open source projects, so there are many institutional eyes that actively contribute to its improvement. IME if we ever encountered an issue at work with ZK, we found it impossible to trace it down to a bug that we could fix and upstream. Etcd’s been easier in this regard.
- lima 6y agoAgreed, etcd is rock solid. With regards to Kafka, it's probably easier and more robust to add their own consensus layer rather than switching to etcd - Kafka is already a distributed system built by a team of distributed systems engineers. It makes sense for them to build their own consensus, deeply integrated with the replication mechanism, rather than relying on an external database.
- ashtonkem 6y agoIt might be an effect of popularity, but I rarely hear complaints about reliability out of Etcd, while I hear about ZK issues on a consistent basis.