4 ms·
Our kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same
by capmorganbih 2y ago
Our kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same jvm. I need data to live through restarts
- sriram_malhar 2y agoCan you explain what your issue is with Kafka? What makes it not reliable enough?
- capmorganbih 2y agoKafka is good. The problem is we don’t have a dedicated person to manage it so sometimes we have kafka outages
- jojohohanon 2y agoThere’s a whole conference in Vegas next week that want to sell you something.
- dikei 2y agoSo basically, you want to build a write-ahead-log before writing data to Kafka, and I think you're underestimating the effort to implement a WAL. If you don't have a person that can manage Kafka, you almost definitely don't have the person to maintain a WAL.
- lizhenqi 2y agoJust a suggestion, you can learn about automq (a message queue that has rewritten the storage layer based on Kafka), which has done a lot in automatic recovery and self-balancing, maybe it will help you. If you don't need it, you can ignore this comment.
- kasey_junk 2y agoYou could but its not purpose built for that. You'd probably be happier using some other memory mapped file format for that.
- capmorganbih 2y agoCould you please suggest java library for this if you know one?
- kasey_junk 2y agohttps://github.com/OpenHFT/Chronicle-Queue?tab=readme-ov-file#what-is-chronicle-queue https://github.com/OpenHFT/Chronicle-Queue?tab=readme-ov-fil...
- avinassh 2y agocrazy idea, but SQLite
- dingi 2y agoWrite your data to a MognoDB database. Then use the Kafka Mongo connector to pull data from MongoDB to Kafka.