4 ms·

Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or

Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.

saurik 6y ago
I feel like this just argues that ZooKeeper should be a library that can be embedded in the servers of other projects (even if it is conceptually a separate server in the same memory space listening on its own ports; but like, it is otherwise entirely hidden inside of the other program and is entirely managed and configured by it also).
- chucky_z 6y ago
  Isn't this exactly what lead to Hashicorp's memberlist library being used all over in random projects?
- skyde 6y ago
  100 % agree we need good (etch/zookeeper) as a library. I wrote something similar but that provide only (lock/lease) using paxos. The problem with raft ETCD and zookeeper replicates state machine design is that : - machine leaving and joining the ensemble dynamically is something hard to do correctly - optimal quorum size is no more than 5, you can setup other node as observer but it’s hard to decide which of 1000 node should be in the quorum ...
  - lima 6y ago
    The etcd server can be embedded in other projects, and their Raft library is used by plenty of other projects like CockroachDB.
    - skyde 6y ago
      yes it can but it’s a real pain compared to hashicorp version. I really hope they clean up the library to have a simple API. This way it could become the standard for GO and we can stop wasting effort on multiple raft implementation.
- sagichmal 6y ago
  Sounds good to me, more things like ZooKeeper should be embeddable.
halbritt 6y ago
The fact that many people are scared away from Kafka simply because they don't want to run zookeeper should be telling.

saurik 6y ago

I feel like this just argues that ZooKeeper should be a library that can be embedded in the servers of other projects (even if it is conceptually a separate server in the same memory space listening on its own ports; but like, it is otherwise entirely hidden inside of the other program and is entirely managed and configured by it also).

chucky_z 6y ago
Isn't this exactly what lead to Hashicorp's memberlist library being used all over in random projects?
skyde 6y ago
100 % agree we need good (etch/zookeeper) as a library. I wrote something similar but that provide only (lock/lease) using paxos. The problem with raft ETCD and zookeeper replicates state machine design is that : - machine leaving and joining the ensemble dynamically is something hard to do correctly - optimal quorum size is no more than 5, you can setup other node as observer but it’s hard to decide which of 1000 node should be in the quorum ...
- lima 6y ago
  The etcd server can be embedded in other projects, and their Raft library is used by plenty of other projects like CockroachDB.
  - skyde 6y ago
    yes it can but it’s a real pain compared to hashicorp version. I really hope they clean up the library to have a simple API. This way it could become the standard for GO and we can stop wasting effort on multiple raft implementation.
sagichmal 6y ago
Sounds good to me, more things like ZooKeeper should be embeddable.

halbritt 6y ago

The fact that many people are scared away from Kafka simply because they don't want to run zookeeper should be telling.