7 ms·

I've already written with one of the core committers to BookKeeper and yes, I think I'll use it. That said, I'm always looking for users and contributors to ht

I've already written with one of the core committers to BookKeeper and yes, I think I'll use it. That said, I'm always looking for users and contributors to https://sirix.io/ https://sirix.io/ or https://github.com/sirixdb/sirix https://github.com/sirixdb/sirix. That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper. I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn some TypeScript along with using Vue.js and D3js), to interact with SirixDB and to build interactive visualizations to compare revisions of JSON- or XML-resources in SirixDB (stored in a binary format of course highly optimized for space-efficient snapshots). I think as I'm lacking users it might be more useful, but for sure I'm at least as eager to put forth the idea of scalable SirixDB databases :-) So, I'd like to use BookKeeper most probably (single writer, read your own write consistency, using synchronous -- for a quorum -- and asynchronous replication for the rest, exactly once semantics...). The thing I don't like is that we also need ZooKeeper, but yeah. BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)

rad_gruchalski 7y ago
> BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-) One of the reasons is that your storage is not tied to ingest. With Kafka, your broker sits where the data sits. That has advantages and disadvantages. The disadvantages are that it's really difficult to do unlimited history topics / partitions in Kafka edit for significant volume topics. You can have only so much storage and only so many disks attached to your machine. It's not possible to add volumes at runtime. I could be wrong but I think Kafka still requires a broker restart after adding volumes to the server properties. With BookKeeper, you just add nodes with more storage. BookKeeper tracks who was the owner of that partition at the time of write. It's possible to spread partitions across multiple machines.
eternalban 7y ago
This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how... > The thing I don't like is that we also need BookKeeper, but yeah. Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.] > sirixdb Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working. Wish you success in your effort!
- atombender 7y ago
  Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!
  - eternalban 7y ago
    To be clear, it is BookKeeper that can alternatively use etcd not Pulsar. https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-store/ https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto... https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed70f6feb02ef14d2d032abb9e https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...
    - atombender 7y ago
      Absolutely. Pulsar looks nice, too, but I think BookKeeper not needing ZK gives it an edge over Kafka. I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"? From what I can tell, BK supports log streaming just like Kafka, so you can use it to build a pub/sub system if you don't need all of Pulsar's features — right?
      - eternalban 7y ago
        Pulsar is built on top of BK. BK has it own consensus mechanism (ZK, etcd), and Pulsar also has its own (low traffic) consensus via ZK. The article cited above does a very good job of addressing the advantages of Pulsar's architecture over Kafka's. Pulsar also provides for user defined functions (that can run on the broker). This is reminiscent of Postgres's User Defined Functions. BK provides ledgers, DistributedLog provides log semantics on top of those ledgers. Pulsar builds a full blown messaging/streaming system on top of that. Per this thread's OP, it seems to make more sense to use BK/DL directly if you want a WAL component to build a DB.

rad_gruchalski 7y ago

> BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-) One of the reasons is that your storage is not tied to ingest. With Kafka, your broker sits where the data sits. That has advantages and disadvantages. The disadvantages are that it's really difficult to do unlimited history topics / partitions in Kafka edit for significant volume topics. You can have only so much storage and only so many disks attached to your machine. It's not possible to add volumes at runtime. I could be wrong but I think Kafka still requires a broker restart after adding volumes to the server properties. With BookKeeper, you just add nodes with more storage. BookKeeper tracks who was the owner of that partition at the time of write. It's possible to spread partitions across multiple machines.

eternalban 7y ago

This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how... > The thing I don't like is that we also need BookKeeper, but yeah. Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.] > sirixdb Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working. Wish you success in your effort!

atombender 7y ago
Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!
- eternalban 7y ago
  To be clear, it is BookKeeper that can alternatively use etcd not Pulsar. https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-store/ https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto... https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed70f6feb02ef14d2d032abb9e https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...
  - atombender 7y ago
    Absolutely. Pulsar looks nice, too, but I think BookKeeper not needing ZK gives it an edge over Kafka. I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"? From what I can tell, BK supports log streaming just like Kafka, so you can use it to build a pub/sub system if you don't need all of Pulsar's features — right?
    - eternalban 7y ago
      Pulsar is built on top of BK. BK has it own consensus mechanism (ZK, etcd), and Pulsar also has its own (low traffic) consensus via ZK. The article cited above does a very good job of addressing the advantages of Pulsar's architecture over Kafka's. Pulsar also provides for user defined functions (that can run on the broker). This is reminiscent of Postgres's User Defined Functions. BK provides ledgers, DistributedLog provides log semantics on top of those ledgers. Pulsar builds a full blown messaging/streaming system on top of that. Per this thread's OP, it seems to make more sense to use BK/DL directly if you want a WAL component to build a DB.