4 ms·

Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM compo

Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!

eternalban 7y ago
To be clear, it is BookKeeper that can alternatively use etcd not Pulsar. https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-store/ https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto... https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed70f6feb02ef14d2d032abb9e https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...
- atombender 7y ago
  Absolutely. Pulsar looks nice, too, but I think BookKeeper not needing ZK gives it an edge over Kafka. I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"? From what I can tell, BK supports log streaming just like Kafka, so you can use it to build a pub/sub system if you don't need all of Pulsar's features — right?
  - eternalban 7y ago
    Pulsar is built on top of BK. BK has it own consensus mechanism (ZK, etcd), and Pulsar also has its own (low traffic) consensus via ZK. The article cited above does a very good job of addressing the advantages of Pulsar's architecture over Kafka's. Pulsar also provides for user defined functions (that can run on the broker). This is reminiscent of Postgres's User Defined Functions. BK provides ledgers, DistributedLog provides log semantics on top of those ledgers. Pulsar builds a full blown messaging/streaming system on top of that. Per this thread's OP, it seems to make more sense to use BK/DL directly if you want a WAL component to build a DB.
    - atombender 7y ago
      Thanks. Right, I know Pulsar is built on top of BK — it's just unclear how much extra value it adds. I'm looking at using BK for a WAL-like component for a document store. It does involve multiple (many!) consumers tailing the log, but unlike your typical Kafka setup, there's need for no partitioning. I was unsure of whether BK provides an API that's nice to use for non-Java languages like Go and Rust, which is what we use. Apache projects written in Java (or Scala), particularly those in the Hadoop ecosystem, tend to be very JVM-centric, and often they're impossible to use from other languages because there's no non-Java API. Kafka is an outlier here, but it certainly started out with a Java-only client and it took years for non-Java clients to mature enough to be usable. I thought I saw that BK had a REST API, but now I see that it's just an admin API. From what I can tell, it's exactly like every other Apache Java project — there's a big monolithic Java client that's tightly coupled with the server, and to write a client in any other language you'd have to reverse-engineer it and essentially port the whole thing, because it's a "fat client" — it looks like it talks directly to ZooKeeper and "bookies", and has a lot of logic built in. The Bookie protocol looks like it's binary and hand-coded (not gRPC or anything nice). So disappointing! /rant Edit: Looks like the files I was looking at are generated from Protobuf. So, yay. Unfortunately, that's just the Bookie protocol. The main client is still fat and in Java.
      - lichtenberger 7y ago
        Regarding the question if Apache Pulsar or Apache BookKeeper for me it's not so clear which of both to use. I think it adds a lot of value if you add the WAL to the public API. Forgot how it's called, I think change data types (CDTs) without searching on google!? So basically you have the stream of events and expose these for use by applications :-)

eternalban 7y ago

To be clear, it is BookKeeper that can alternatively use etcd not Pulsar. https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-store/ https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto... https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed70f6feb02ef14d2d032abb9e https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...

atombender 7y ago
Absolutely. Pulsar looks nice, too, but I think BookKeeper not needing ZK gives it an edge over Kafka. I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"? From what I can tell, BK supports log streaming just like Kafka, so you can use it to build a pub/sub system if you don't need all of Pulsar's features — right?
- eternalban 7y ago
  Pulsar is built on top of BK. BK has it own consensus mechanism (ZK, etcd), and Pulsar also has its own (low traffic) consensus via ZK. The article cited above does a very good job of addressing the advantages of Pulsar's architecture over Kafka's. Pulsar also provides for user defined functions (that can run on the broker). This is reminiscent of Postgres's User Defined Functions. BK provides ledgers, DistributedLog provides log semantics on top of those ledgers. Pulsar builds a full blown messaging/streaming system on top of that. Per this thread's OP, it seems to make more sense to use BK/DL directly if you want a WAL component to build a DB.
  - atombender 7y ago
    Thanks. Right, I know Pulsar is built on top of BK — it's just unclear how much extra value it adds. I'm looking at using BK for a WAL-like component for a document store. It does involve multiple (many!) consumers tailing the log, but unlike your typical Kafka setup, there's need for no partitioning. I was unsure of whether BK provides an API that's nice to use for non-Java languages like Go and Rust, which is what we use. Apache projects written in Java (or Scala), particularly those in the Hadoop ecosystem, tend to be very JVM-centric, and often they're impossible to use from other languages because there's no non-Java API. Kafka is an outlier here, but it certainly started out with a Java-only client and it took years for non-Java clients to mature enough to be usable. I thought I saw that BK had a REST API, but now I see that it's just an admin API. From what I can tell, it's exactly like every other Apache Java project — there's a big monolithic Java client that's tightly coupled with the server, and to write a client in any other language you'd have to reverse-engineer it and essentially port the whole thing, because it's a "fat client" — it looks like it talks directly to ZooKeeper and "bookies", and has a lot of logic built in. The Bookie protocol looks like it's binary and hand-coded (not gRPC or anything nice). So disappointing! /rant Edit: Looks like the files I was looking at are generated from Protobuf. So, yay. Unfortunately, that's just the Bookie protocol. The main client is still fat and in Java.
    - lichtenberger 7y ago
      Regarding the question if Apache Pulsar or Apache BookKeeper for me it's not so clear which of both to use. I think it adds a lot of value if you add the WAL to the public API. Forgot how it's called, I think change data types (CDTs) without searching on google!? So basically you have the stream of events and expose these for use by applications :-)