6 ms·
I’m so happy to see that someone has finally built such thing. I’ve been asking for tiered storage in Kafka since 2016 (https://gruchalski.com/posts/2016-05-08-
by rad_gruchalski 3y ago
I’m so happy to see that someone has finally built such thing. I’ve been asking for tiered storage in Kafka since 2016 (https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-cold-storage/ https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-c...) but this approach is so much better.
Kafka itself could solve this with tiered storage OR, at least, allowing adding volumes at runtime. But neither is possible.
Very cool, have to find some time to take this for a test drive.
- richieartoul 3y ago(WarpStream co-founder) Thanks for the kind words! I'll just add that tiered storage is not quite the same thing because it means you still have to manage local disks and replication carefully, however briefly. Please reach out with questions any time!
- rad_gruchalski 3y agoIndeed. The way I was thinking about it was to implement a different file system for it.
- Sphax 3y agoJust fyi, tiered storage is coming at some point in Kafka: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=97554472#KIP405:KafkaTieredStorage-Status https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...
- rad_gruchalski 3y agoIt’s been like that for a while.
- EdwardDiego 3y agoThe Uber devs are working hard at it.
- EdwardDiego 3y agoI can't edit the above, but it's actually AirBnB.