5 ms·

From what I can tell, Kafka isn't designed for long term data storage. RDBMS systems are designed for this. Kafka is more for streaming data and events, so I'd

From what I can tell, Kafka isn't designed for long term data storage. RDBMS systems are designed for this. Kafka is more for streaming data and events, so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future. Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config to revert back to this behaviour. RDBMS won't delete things unless you tell it to more explicitly. While I think storing things in Kafka is fine generally, there's no way I'd not have a more perminant store of the data somewhere so that I can recreate the Kafka data store if I need to. I'm not sure why they're not just using a boring old DB for that purpose. Perhaps they have a reason, but it's not obvious to me.

deleted 9y ago
[deleted]
miguno 9y ago
Kafka can perfectly keep your data around forever. The only limitation is available disk space (and databases have the same limitation). I'm not implying that it is always the best idea to use Kafka as a long-term storage solution, but likewise a database isn't the silver bullet here either. > so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future Kafka doesn't "tidy up" your data unless it is configured to do so. What's true is that, by default, Kafka will keep your data around for a week "only" but that's a configuration. And most people change it to whatever fits their use case (some lower it to a few hours, some increase it to months or years; others configure it to keep data around forever, typically in combination with Kafka's so-called "log compaction" functionality). > Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config to revert back to this behaviour. RDBMS won't delete things unless you tell it to more explicitly. The DBAs I worked with would now say "Hold my beer..." ;-) > While I think storing things in Kafka is fine generally, there's no way I'd not have a more perminant store of the data somewhere so that I can recreate the Kafka data store if I need to. What's interesting is that more and more users (from my experience) are actually beginning to treat Kafka as the source of truth, and rather recreate other data stores -- like RDBMS, Elastic indexes, etc. -- from it. If you like RDBMS, you can think of Kafka as the DB's transaction log. IMHO a lot of these discussion is about personal preferences, the situation that you are in (cough legacy cough), team skills, etc. There are often good reasons to use Kafka rather than RDBMS (in this context) but also vice versa, or completely different technologies of course (like blob stores, e.g. S3).
- cookiecaper 9y ago
  >Kafka can perfectly keep your data around forever. In the sense that you can fiddle with it to the point where it doesn't purge things automatically, sure. But RDBMS provides more than the promise that it won't delete your data after a set period of time. If that was all we needed, any filesystem from the last 3 decades would serve fine as a "permanent datastore". MySQL has gone through a lot of grief to get to the point where it's safe out of the box. This is important. There are many conceivable situations where an unsafe default can accidentally get worked back into things. At least MySQL has never had a default that truncated databases (afaik)... >The DBAs I worked with would now say "Hold my beer..." ;-) No one's claiming that RDBMS are invincible, but they do provide numerous features specifically designed to minimize data loss and corruption. You can read more about these in the documentation of any major RDBMS. I've mentioned some specifically elsewhere in this thread, but don't want to keep repeating the same things. >IMHO a lot of these discussion is about personal preferences, the situation that you are in (cough legacy cough), team skills, etc. No offense, but this just shows your ignorance of the functionality that you're leaving on the table by considering SQL a cough-legacy-cough solution. We discussed above that "can be configured to not delete things despite defaults and assumptions" is much different from "safe long-term data storage is a major design concern". A messaging queue does not and should not have the same semantics, because it's not intended to keep decades' worth of data, even though it may be possible to avoid the database's eviction routine. We don't have to abuse something just because the developers haven't gone to lengths to override inadvisable configurations.
  - tlberglund 9y ago
    It may be unfair to describe setting a documented configuration parameter as "fiddling." Retention is seven days by default. It is trivial to set it to arbitrarily long periods of time. To my knowledge, this functionality isn't really in question. Whether logs are a good unifying abstraction on which to build systems is in dispute among reasonable people, but whether Kafka randomly deletes stuff is not. :)
    - cookiecaper 9y ago
mayank 9y ago
You've really just laid out feelings rather than concrete technical reasons for why Kafka can't function as a permanent datastore. Can you point to specific design elements in Kafka that would lead you to conclude that it isn't suitable for permanent data storage? Also, Kafka doesn't "do" anything to your old data if you don't want it to. It's also open-source, so these behaviors can be verified.
- pram 9y ago
  I'd say it isn't appropriate for permanent data storage because the individual brokers don't scale well with the amount of logs present. If you have hundreds of partitions, and millions of logs, then any operation dealing with the indexes (like an unclean startup) will take an extremely long amount of time. So your individual brokers are now down for an hour if they don't shut down cleanly. Which happens often (oom, weird zk errors, etc) It scales linearly.
  - LgWoodenBadger 9y ago
    Most of the use-cases you describe wouldn't be resolved by re-reading the entirety of the topics from the beginning of time. They'd resume from their last committed offset and continue where they left off. If you do have to catastrophically recover from the beginning of time, then sure you'd have a rough time. But that's true for any system that would have to do that. It's not Kafka-specific. Now if your consumer was entirely in-memory and made no use of persistent storage itself and it had to recover from the beginning of time, then I'd say that type of problem is your own architectural failure and I have yet to come across a tool, pattern, framework, or architecture that bullet-proofs your foot.
    - pram 9y ago
      Reading the topic from the beginning is not what I'm talking about.

deleted 9y ago

[deleted]

miguno 9y ago

Kafka can perfectly keep your data around forever. The only limitation is available disk space (and databases have the same limitation). I'm not implying that it is always the best idea to use Kafka as a long-term storage solution, but likewise a database isn't the silver bullet here either. > so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future Kafka doesn't "tidy up" your data unless it is configured to do so. What's true is that, by default, Kafka will keep your data around for a week "only" but that's a configuration. And most people change it to whatever fits their use case (some lower it to a few hours, some increase it to months or years; others configure it to keep data around forever, typically in combination with Kafka's so-called "log compaction" functionality). > Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config to revert back to this behaviour. RDBMS won't delete things unless you tell it to more explicitly. The DBAs I worked with would now say "Hold my beer..." ;-) > While I think storing things in Kafka is fine generally, there's no way I'd not have a more perminant store of the data somewhere so that I can recreate the Kafka data store if I need to. What's interesting is that more and more users (from my experience) are actually beginning to treat Kafka as the source of truth, and rather recreate other data stores -- like RDBMS, Elastic indexes, etc. -- from it. If you like RDBMS, you can think of Kafka as the DB's transaction log. IMHO a lot of these discussion is about personal preferences, the situation that you are in (cough legacy cough), team skills, etc. There are often good reasons to use Kafka rather than RDBMS (in this context) but also vice versa, or completely different technologies of course (like blob stores, e.g. S3).

cookiecaper 9y ago
>Kafka can perfectly keep your data around forever. In the sense that you can fiddle with it to the point where it doesn't purge things automatically, sure. But RDBMS provides more than the promise that it won't delete your data after a set period of time. If that was all we needed, any filesystem from the last 3 decades would serve fine as a "permanent datastore". MySQL has gone through a lot of grief to get to the point where it's safe out of the box. This is important. There are many conceivable situations where an unsafe default can accidentally get worked back into things. At least MySQL has never had a default that truncated databases (afaik)... >The DBAs I worked with would now say "Hold my beer..." ;-) No one's claiming that RDBMS are invincible, but they do provide numerous features specifically designed to minimize data loss and corruption. You can read more about these in the documentation of any major RDBMS. I've mentioned some specifically elsewhere in this thread, but don't want to keep repeating the same things. >IMHO a lot of these discussion is about personal preferences, the situation that you are in (cough legacy cough), team skills, etc. No offense, but this just shows your ignorance of the functionality that you're leaving on the table by considering SQL a cough-legacy-cough solution. We discussed above that "can be configured to not delete things despite defaults and assumptions" is much different from "safe long-term data storage is a major design concern". A messaging queue does not and should not have the same semantics, because it's not intended to keep decades' worth of data, even though it may be possible to avoid the database's eviction routine. We don't have to abuse something just because the developers haven't gone to lengths to override inadvisable configurations.
- tlberglund 9y ago
  It may be unfair to describe setting a documented configuration parameter as "fiddling." Retention is seven days by default. It is trivial to set it to arbitrarily long periods of time. To my knowledge, this functionality isn't really in question. Whether logs are a good unifying abstraction on which to build systems is in dispute among reasonable people, but whether Kafka randomly deletes stuff is not. :)
  - cookiecaper 9y ago

mayank 9y ago

You've really just laid out feelings rather than concrete technical reasons for why Kafka can't function as a permanent datastore. Can you point to specific design elements in Kafka that would lead you to conclude that it isn't suitable for permanent data storage? Also, Kafka doesn't "do" anything to your old data if you don't want it to. It's also open-source, so these behaviors can be verified.

pram 9y ago
I'd say it isn't appropriate for permanent data storage because the individual brokers don't scale well with the amount of logs present. If you have hundreds of partitions, and millions of logs, then any operation dealing with the indexes (like an unclean startup) will take an extremely long amount of time. So your individual brokers are now down for an hour if they don't shut down cleanly. Which happens often (oom, weird zk errors, etc) It scales linearly.
- LgWoodenBadger 9y ago
  Most of the use-cases you describe wouldn't be resolved by re-reading the entirety of the topics from the beginning of time. They'd resume from their last committed offset and continue where they left off. If you do have to catastrophically recover from the beginning of time, then sure you'd have a rough time. But that's true for any system that would have to do that. It's not Kafka-specific. Now if your consumer was entirely in-memory and made no use of persistent storage itself and it had to recover from the beginning of time, then I'd say that type of problem is your own architectural failure and I have yet to come across a tool, pattern, framework, or architecture that bullet-proofs your foot.
  - pram 9y ago
    Reading the topic from the beginning is not what I'm talking about.