3 ms·
You've really just laid out feelings rather than concrete technical reasons for why Kafka can't function as a permanent datastore. Can you point to specific de
by mayank 9y ago
You've really just laid out feelings rather than concrete technical reasons for why Kafka can't function as a permanent datastore.
Can you point to specific design elements in Kafka that would lead you to conclude that it isn't suitable for permanent data storage?
Also, Kafka doesn't "do" anything to your old data if you don't want it to. It's also open-source, so these behaviors can be verified.
- pram 9y agoI'd say it isn't appropriate for permanent data storage because the individual brokers don't scale well with the amount of logs present. If you have hundreds of partitions, and millions of logs, then any operation dealing with the indexes (like an unclean startup) will take an extremely long amount of time. So your individual brokers are now down for an hour if they don't shut down cleanly. Which happens often (oom, weird zk errors, etc) It scales linearly.
- LgWoodenBadger 9y agoMost of the use-cases you describe wouldn't be resolved by re-reading the entirety of the topics from the beginning of time. They'd resume from their last committed offset and continue where they left off. If you do have to catastrophically recover from the beginning of time, then sure you'd have a rough time. But that's true for any system that would have to do that. It's not Kafka-specific. Now if your consumer was entirely in-memory and made no use of persistent storage itself and it had to recover from the beginning of time, then I'd say that type of problem is your own architectural failure and I have yet to come across a tool, pattern, framework, or architecture that bullet-proofs your foot.
- pram 9y agoReading the topic from the beginning is not what I'm talking about.