6 ms·
I've wondered for a while now why there isn't something like a cross between Kafka (durable queue broker) and Datomic (a "database" partitioned into storage chu
by derefr 7y ago
I've wondered for a while now why there isn't something like a cross between Kafka (durable queue broker) and Datomic (a "database" partitioned into storage chunks, where each chunk canonically lives in object storage and is only cached by the instances doing the queries.)
I'd love to see a queuing infrastructure that consists of two parallel clusters, running on the same machines:
• a set of write nodes, which each "own" partitions on a topic (in the sense of being that partition's sole writer); where a write-node receives writes to its partition until it fills up a chunk, at which point it shoves the chunk into object storage;
• a set of read nodes, which serve reads and/or streaming subscription requests by 1. streaming archival data by retrieving chunks from object storage into a read-node-local MRU-on-disk-pressure cache; and then 2. once the read stream "catches up" to the end of the archival data, it would switch to proxying to the relevant write-node, which would serve the query/stream from its uncommitted chunk.
-----
I feel like this queue architecture would be useful for all sorts of things, at least for my own use-cases. Especially because it'd scale perfectly-well down to running one read-node and one write-node on one machine, while still being able to make durability guarantees about the committed data. (The durability of uncommitted data, I don't much need to worry much about, personally.)
Is there anything like this on the market?
- rad_gruchalski 7y agoMaybe Pulsar gets close. Your general question regarding Kafka reminds me of this: https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32929d0a57b2 https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32... I’ve written that. Edit: kafka jira did not get far: https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-3726 https://issues.apache.org/jira/plugins/servlet/mobile#issue/...
- PhoenixReborn 7y agoConfluent actually has released "tiered storage" which appears to be a similar feature to what Pulsar has. https://www.confluent.io/blog/infinite-kafka-storage-in-confluent-platform/ https://www.confluent.io/blog/infinite-kafka-storage-in-conf... It's still up in the air as to whether this feature will make its way into the open-source Kafka though.
- jhgg 7y agoYou are looking for Druid: http://druid.apache.org/ http://druid.apache.org/
- pram 7y agoIf the read node ultimately has a dependency on the write node, whats the point of separating them in your architecture?