5 ms·

I like the idea, to use Apache Kafka as the underlying log. It's in principle what Martin Kleppmann suggests. As I also want to use a distributed log in the fu

I like the idea, to use Apache Kafka as the underlying log. It's in principle what Martin Kleppmann suggests. As I also want to use a distributed log in the future: Do you know Apache Pulsar or the underlying BookKeeper, which I think was especially made for a distributed WAL? I might either want to use Apache BookKeeper or Apache Pulsar for a distributed log for scaling my Open Source temporal database, too. Furthermore I'd like to expose the API for streaming changes into the Browser or wherever you want to :-)

eternalban 7y ago
I've been itching for a while to build something on top of BookKeeper's DistributedLog. I suggest ditching the Pulsar and just using the DL.
- lichtenberger 7y ago
  I've already written with one of the core committers to BookKeeper and yes, I think I'll use it. That said, I'm always looking for users and contributors to https://sirix.io/ https://sirix.io/ or https://github.com/sirixdb/sirix https://github.com/sirixdb/sirix. That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper. I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn some TypeScript along with using Vue.js and D3js), to interact with SirixDB and to build interactive visualizations to compare revisions of JSON- or XML-resources in SirixDB (stored in a binary format of course highly optimized for space-efficient snapshots). I think as I'm lacking users it might be more useful, but for sure I'm at least as eager to put forth the idea of scalable SirixDB databases :-) So, I'd like to use BookKeeper most probably (single writer, read your own write consistency, using synchronous -- for a quorum -- and asynchronous replication for the rest, exactly once semantics...). The thing I don't like is that we also need ZooKeeper, but yeah. BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)
  - rad_gruchalski 7y ago
    > BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-) One of the reasons is that your storage is not tied to ingest. With Kafka, your broker sits where the data sits. That has advantages and disadvantages. The disadvantages are that it's really difficult to do unlimited history topics / partitions in Kafka edit for significant volume topics. You can have only so much storage and only so many disks attached to your machine. It's not possible to add volumes at runtime. I could be wrong but I think Kafka still requires a broker restart after adding volumes to the server properties. With BookKeeper, you just add nodes with more storage. BookKeeper tracks who was the owner of that partition at the time of write. It's possible to spread partitions across multiple machines.
  - eternalban 7y ago
    This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how... > The thing I don't like is that we also need BookKeeper, but yeah. Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.] > sirixdb Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working. Wish you success in your effort!
    - atombender 7y ago
      Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!
      - eternalban 7y ago
        To be clear, it is BookKeeper that can alternatively use etcd not Pulsar. https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-store/ https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto... https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed70f6feb02ef14d2d032abb9e https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...

eternalban 7y ago

I've been itching for a while to build something on top of BookKeeper's DistributedLog. I suggest ditching the Pulsar and just using the DL.

lichtenberger 7y ago
I've already written with one of the core committers to BookKeeper and yes, I think I'll use it. That said, I'm always looking for users and contributors to https://sirix.io/ https://sirix.io/ or https://github.com/sirixdb/sirix https://github.com/sirixdb/sirix. That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper. I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn some TypeScript along with using Vue.js and D3js), to interact with SirixDB and to build interactive visualizations to compare revisions of JSON- or XML-resources in SirixDB (stored in a binary format of course highly optimized for space-efficient snapshots). I think as I'm lacking users it might be more useful, but for sure I'm at least as eager to put forth the idea of scalable SirixDB databases :-) So, I'd like to use BookKeeper most probably (single writer, read your own write consistency, using synchronous -- for a quorum -- and asynchronous replication for the rest, exactly once semantics...). The thing I don't like is that we also need ZooKeeper, but yeah. BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)
- rad_gruchalski 7y ago
  > BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-) One of the reasons is that your storage is not tied to ingest. With Kafka, your broker sits where the data sits. That has advantages and disadvantages. The disadvantages are that it's really difficult to do unlimited history topics / partitions in Kafka edit for significant volume topics. You can have only so much storage and only so many disks attached to your machine. It's not possible to add volumes at runtime. I could be wrong but I think Kafka still requires a broker restart after adding volumes to the server properties. With BookKeeper, you just add nodes with more storage. BookKeeper tracks who was the owner of that partition at the time of write. It's possible to spread partitions across multiple machines.
- eternalban 7y ago
  This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works https://jack-vanlightly.com/blog/2018/10/2/understanding-how... > The thing I don't like is that we also need BookKeeper, but yeah. Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.] > sirixdb Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working. Wish you success in your effort!
  - atombender 7y ago
    Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!
    - eternalban 7y ago
      To be clear, it is BookKeeper that can alternatively use etcd not Pulsar. https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-store/ https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto... https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed70f6feb02ef14d2d032abb9e https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...