3 ms·

I expect Pulsar to stay with Zookeeper. Kafka currently stores topic and partition info on ZK ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+str

I expect Pulsar to stay with Zookeeper. Kafka currently stores topic and partition info on ZK ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data... ), which can get to be a lot of data. But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable.

manigandham 6y ago
True, Pulsar's architecture is more reliable and scalable than Kafka, but it would still be nice to remove any extra dependencies.
biggestlou 6y ago
That is correct: Pulsar uses ZooKeeper relatively lightly and only for things strictly necessary for the cluster to run. Everything else (message data, topic info, cursors, schemas) is stored in BookKeeper. Would still be great to see ZooKeeper made superfluous, though.
colin_mccabe 6y ago
But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable. Unfortunately, this is not correct. BookKeeper stores a lot of information in ZookKeeper. By extension, Pulsar (which is based on BookKeeper) also stores a lot of metadata there as well. For example, from the BK documentation ( https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.html https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.h... ): An application first creates a ledger before writing to bookies through a local BookKeeper client instance. Upon creating a ledger, a BookKeeper client writes metadata about the ledger to ZooKeeper. Each ledger currently has a single writer. This writer has to execute a close ledger operation before any other client can read from it. If the writer of a ledger does not close a ledger properly because, for example, it has crashed before having the opportunity of closing the ledger, then the next client that tries to open a ledger executes a procedure to recover it. As closing a ledger consists essentially of writing the last entry written to a ledger to ZooKeeper, the recovery procedure simply finds the last entry written correctly and writes it to ZooKeeper.
rohitshekhar 6y ago
The bookKeeper has a nontrivial dependency on the ZK. Sooner or later Pulsar needs to solve the same issue.

manigandham 6y ago

True, Pulsar's architecture is more reliable and scalable than Kafka, but it would still be nice to remove any extra dependencies.

biggestlou 6y ago

That is correct: Pulsar uses ZooKeeper relatively lightly and only for things strictly necessary for the cluster to run. Everything else (message data, topic info, cursors, schemas) is stored in BookKeeper. Would still be great to see ZooKeeper made superfluous, though.

colin_mccabe 6y ago

But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable. Unfortunately, this is not correct. BookKeeper stores a lot of information in ZookKeeper. By extension, Pulsar (which is based on BookKeeper) also stores a lot of metadata there as well. For example, from the BK documentation ( https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.html https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.h... ): An application first creates a ledger before writing to bookies through a local BookKeeper client instance. Upon creating a ledger, a BookKeeper client writes metadata about the ledger to ZooKeeper. Each ledger currently has a single writer. This writer has to execute a close ledger operation before any other client can read from it. If the writer of a ledger does not close a ledger properly because, for example, it has crashed before having the opportunity of closing the ledger, then the next client that tries to open a ledger executes a procedure to recover it. As closing a ledger consists essentially of writing the last entry written to a ledger to ZooKeeper, the recovery procedure simply finds the last entry written correctly and writes it to ZooKeeper.

rohitshekhar 6y ago

The bookKeeper has a nontrivial dependency on the ZK. Sooner or later Pulsar needs to solve the same issue.