7 ms·

>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH. Cue the facepalm, head in hands, etc.

>We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch. SIGH. Cue the facepalm, head in hands, etc. I'm not going to get into a big thing here. But if you find yourself saying "I need to keep this thing forever no matter what" and then you try to use something that even entertains the notion of automatic eviction/deletion semantics as the system of record, you're doing it wrong. Not to burst the bubble of the techno-hipsters, but Kafka is "durable" relative to message brokers like RabbitMQ, not relative to a system actually designed to store decades of mission-critical data. Those systems are called "RDBMS". Elsewhere in the article he says that they have less than 100GB of data and that it's mostly text. This is massive overarchitecture that isn't even covering the basic flanks that it thinks it is, such as data permanence. I would really like to read the article that discusses why Postgres or MySQL couldn't have served this purpose equally well.

bognition 9y ago
OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.
- cachemiss 9y ago
  That's a bit of an oversimplification. Production grade RDBMS systems have far more guard rails, testing and work put in to them than Kafka. It's relatively straight forward to lose data in Kafka, I've done it (its usually control plane bugs, not data plane).
- tepe1 9y ago
  GP is clueless. Kafka is in fact more resilient and much more performant than systems like MySQL. It's the difference between having one writer and many writers. More importantly, adding yet another database to an enterprise that's already overrun databases is not going to fix anything. Systems like Kafka are deployed precisely because there are already many domain-specific databases at work and now there's a need to tie them together. You can't have a single database that works for all domains, for everything from elastic search text to billing. That's just dumb. So you have many databases and tie them together using a transactional event log. BTW, the NYTimes, like virtually any mature enterprise, probably has a robust data warehouse and data retention strategy. I kind of doubt they're in the habit of losing data given their archives go back more than a century. But it's important to distinguish front-office transactional concerns (actually handing real-time requests from content producers and consumers) from back-office concerns (reporting, back-ups). They are very different domains as many enterprises are slowly discovering.
- cookiecaper 9y ago
  In Postgres, I never have to worry that the server will be accidentally loaded with `retention.bytes` or `retention.days` set too low and, as a result, choose to delete everything in the database, generating a wholly artificial "disaster" that can result in long periods of disruption or downtime (at a minimum; worst case is permanent data loss). It is true that someone could issue `DROP DATABASE`, `rm -rf` the filesystem on the database server, or so forth, so my point is not that other systems are invincible. It's just that a properly-configured RDBMS is designed to take data integrity extremely seriously and provides numerous failsafes and protective mechanisms to try to ensure that any data "loss" is absolutely intentional. On a RDBMS, things like foreign key constraints prevent deletion of dependent records, mature and well-defined access control systems prevent accidental or malicious record alteration, concurrency models and transactions keep data in a good state, etc. Kafka, on the other hand, is designed to automatically and silently delete/purge data whenever a couple of flags are flipped. That is not a flaw in Kafka itself; it's designed to do that so that you don't have to interrupt your day and purge expired/old/processed data all the time. It's a flaw in architectures that misinterpret Kafka's log paradigm as a replacement for a real data storage/retrieval/archive system. I've had this argument countless times with people who've tried to use RabbitMQ as a system of record (if only for a few minutes while the messages sat in queue). There's just some fundamental disconnect for a lot of developers where they don't understand that something accepting the handoff doesn't mean that the data is inherently safe.
  - tlberglund 9y ago
    I don't mean to be cavalier about misconfiguration, but it's not like the retention period is the Sword of Damocles. It's a configuration setting, and Kafka honors it reliably. As other commenters have pointed out, there are other bad things you can do to cause data loss with any system no matter how hard you try. These stories will continue to grace post-mortem blog posts long after we are gone, but stories of Kafka accidentally not retaining data do not seem to be thick on the ground. Any non-trivial system has its rough edges, but this just doesn't seem to be one of them for Kafka.
  - skybrian 9y ago
    So, remove the flag that you'll never use and recompile? I'm not sure this is worse than using a Unix box with a working "rm" command.
- scaryclam 9y ago
  From what I can tell, Kafka isn't designed for long term data storage. RDBMS systems are designed for this. Kafka is more for streaming data and events, so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future. Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config to revert back to this behaviour. RDBMS won't delete things unless you tell it to more explicitly. While I think storing things in Kafka is fine generally, there's no way I'd not have a more perminant store of the data somewhere so that I can recreate the Kafka data store if I need to. I'm not sure why they're not just using a boring old DB for that purpose. Perhaps they have a reason, but it's not obvious to me.
  - deleted 9y ago
    [deleted]
  - miguno 9y ago
    Kafka can perfectly keep your data around forever. The only limitation is available disk space (and databases have the same limitation). I'm not implying that it is always the best idea to use Kafka as a long-term storage solution, but likewise a database isn't the silver bullet here either. > so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future Kafka doesn't "tidy up" your data unless it is configured to do so. What's true is that, by default, Kafka will keep your data around for a week "only" but that's a configuration. And most people change it to whatever fits their use case (some lower it to a few hours, some increase it to months or years; others configure it to keep data around forever, typically in combination with Kafka's so-called "log compaction" functionality). > Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config to revert back to this behaviour. RDBMS won't delete things unless you tell it to more explicitly. The DBAs I worked with would now say "Hold my beer..." ;-) > While I think storing things in Kafka is fine generally, there's no way I'd not have a more perminant store of the data somewhere so that I can recreate the Kafka data store if I need to. What's interesting is that more and more users (from my experience) are actually beginning to treat Kafka as the source of truth, and rather recreate other data stores -- like RDBMS, Elastic indexes, etc. -- from it. If you like RDBMS, you can think of Kafka as the DB's transaction log. IMHO a lot of these discussion is about personal preferences, the situation that you are in (cough legacy cough), team skills, etc. There are often good reasons to use Kafka rather than RDBMS (in this context) but also vice versa, or completely different technologies of course (like blob stores, e.g. S3).
robohamburger 9y ago
I skimmed the article but I imagined they were using it as a secondary data store. I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically).
- Diederich 9y ago
  "With the log as the source of truth, there is no longer any need for a single database that all systems have to use. Instead, every system can create its own data store (database) – its own materialized view – representing only the data it needs, in the form that is the most useful for that system. This massively simplifies the role of databases in an architecture, and makes them more suited to the need of each application."
  - sidlls 9y ago
    And then you end up with a different flavor of data store for every team, complete with its own idioms and (probably duplicative) business logic. Unless their devops/SRE staff is up to the task this "architecture" is a nightmare waiting to happen.
  - robohamburger 9y ago
    Fair enough. It seems like it still ought to be able to burn the kafka+elasticsearch world down and submit everything to kafka with such a setup (and thus elasticsearch). I would certainly not sleep very well at night if I could not.
- miguno 9y ago
  > I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically). That (replaying if needed) is exactly what Kafka allows you to do, unless I misunderstood what you wrote.
  - robohamburger 9y ago
_lex 9y ago
I think the POV he's taken is that the kafka stream is the one true datasource (for all time), with all other dbs being derivatives. This insane strategy seems to be over engineered to get around db migrations... though i'm sure the event stream will also change over time, and he'll have to write migration-like code anyways.
- rhizome 9y ago
  If datasource engineers are for some reason implementing "ontogeny recapitulates phylogeny," they won't be creating migration-like code, they'll be writing ETL filters. Break out your Members Only jackets!
- LgWoodenBadger 9y ago
  There's not just one reason for it. Kafka lets dependent consumers transform that data into whatever model is appropriate for their use case. There's no one-size-fits-all E-R model for data for all use cases. It's not just working around db migrations. It's also providing you with the ability to model the data as many ways as you require. How else would you do it, materialized views? How often are they materialized? Regular views? How performant are they? There are many benefits to this approach, and a lot of them require a different way of thinking. It's a different paradigm.
  - dreamfactored 9y ago
    Isn't that just skinny model, fat client?

bognition 9y ago

OOC what makes a RDBMS more durable that a Kafka? Both of them are systems for representing data on disk. I'd love to hear why one representation system is better at disaster recovery than another.

cachemiss 9y ago
That's a bit of an oversimplification. Production grade RDBMS systems have far more guard rails, testing and work put in to them than Kafka. It's relatively straight forward to lose data in Kafka, I've done it (its usually control plane bugs, not data plane).
tepe1 9y ago
GP is clueless. Kafka is in fact more resilient and much more performant than systems like MySQL. It's the difference between having one writer and many writers. More importantly, adding yet another database to an enterprise that's already overrun databases is not going to fix anything. Systems like Kafka are deployed precisely because there are already many domain-specific databases at work and now there's a need to tie them together. You can't have a single database that works for all domains, for everything from elastic search text to billing. That's just dumb. So you have many databases and tie them together using a transactional event log. BTW, the NYTimes, like virtually any mature enterprise, probably has a robust data warehouse and data retention strategy. I kind of doubt they're in the habit of losing data given their archives go back more than a century. But it's important to distinguish front-office transactional concerns (actually handing real-time requests from content producers and consumers) from back-office concerns (reporting, back-ups). They are very different domains as many enterprises are slowly discovering.
cookiecaper 9y ago
In Postgres, I never have to worry that the server will be accidentally loaded with `retention.bytes` or `retention.days` set too low and, as a result, choose to delete everything in the database, generating a wholly artificial "disaster" that can result in long periods of disruption or downtime (at a minimum; worst case is permanent data loss). It is true that someone could issue `DROP DATABASE`, `rm -rf` the filesystem on the database server, or so forth, so my point is not that other systems are invincible. It's just that a properly-configured RDBMS is designed to take data integrity extremely seriously and provides numerous failsafes and protective mechanisms to try to ensure that any data "loss" is absolutely intentional. On a RDBMS, things like foreign key constraints prevent deletion of dependent records, mature and well-defined access control systems prevent accidental or malicious record alteration, concurrency models and transactions keep data in a good state, etc. Kafka, on the other hand, is designed to automatically and silently delete/purge data whenever a couple of flags are flipped. That is not a flaw in Kafka itself; it's designed to do that so that you don't have to interrupt your day and purge expired/old/processed data all the time. It's a flaw in architectures that misinterpret Kafka's log paradigm as a replacement for a real data storage/retrieval/archive system. I've had this argument countless times with people who've tried to use RabbitMQ as a system of record (if only for a few minutes while the messages sat in queue). There's just some fundamental disconnect for a lot of developers where they don't understand that something accepting the handoff doesn't mean that the data is inherently safe.
- tlberglund 9y ago
  I don't mean to be cavalier about misconfiguration, but it's not like the retention period is the Sword of Damocles. It's a configuration setting, and Kafka honors it reliably. As other commenters have pointed out, there are other bad things you can do to cause data loss with any system no matter how hard you try. These stories will continue to grace post-mortem blog posts long after we are gone, but stories of Kafka accidentally not retaining data do not seem to be thick on the ground. Any non-trivial system has its rough edges, but this just doesn't seem to be one of them for Kafka.
- skybrian 9y ago
  So, remove the flag that you'll never use and recompile? I'm not sure this is worse than using a Unix box with a working "rm" command.
scaryclam 9y ago
From what I can tell, Kafka isn't designed for long term data storage. RDBMS systems are designed for this. Kafka is more for streaming data and events, so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future. Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config to revert back to this behaviour. RDBMS won't delete things unless you tell it to more explicitly. While I think storing things in Kafka is fine generally, there's no way I'd not have a more perminant store of the data somewhere so that I can recreate the Kafka data store if I need to. I'm not sure why they're not just using a boring old DB for that purpose. Perhaps they have a reason, but it's not obvious to me.
- deleted 9y ago
  [deleted]
- miguno 9y ago
  Kafka can perfectly keep your data around forever. The only limitation is available disk space (and databases have the same limitation). I'm not implying that it is always the best idea to use Kafka as a long-term storage solution, but likewise a database isn't the silver bullet here either. > so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future Kafka doesn't "tidy up" your data unless it is configured to do so. What's true is that, by default, Kafka will keep your data around for a week "only" but that's a configuration. And most people change it to whatever fits their use case (some lower it to a few hours, some increase it to months or years; others configure it to keep data around forever, typically in combination with Kafka's so-called "log compaction" functionality). > Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config to revert back to this behaviour. RDBMS won't delete things unless you tell it to more explicitly. The DBAs I worked with would now say "Hold my beer..." ;-) > While I think storing things in Kafka is fine generally, there's no way I'd not have a more perminant store of the data somewhere so that I can recreate the Kafka data store if I need to. What's interesting is that more and more users (from my experience) are actually beginning to treat Kafka as the source of truth, and rather recreate other data stores -- like RDBMS, Elastic indexes, etc. -- from it. If you like RDBMS, you can think of Kafka as the DB's transaction log. IMHO a lot of these discussion is about personal preferences, the situation that you are in (cough legacy cough), team skills, etc. There are often good reasons to use Kafka rather than RDBMS (in this context) but also vice versa, or completely different technologies of course (like blob stores, e.g. S3).

robohamburger 9y ago

I skimmed the article but I imagined they were using it as a secondary data store. I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically).

Diederich 9y ago
"With the log as the source of truth, there is no longer any need for a single database that all systems have to use. Instead, every system can create its own data store (database) – its own materialized view – representing only the data it needs, in the form that is the most useful for that system. This massively simplifies the role of databases in an architecture, and makes them more suited to the need of each application."
- sidlls 9y ago
  And then you end up with a different flavor of data store for every team, complete with its own idioms and (probably duplicative) business logic. Unless their devops/SRE staff is up to the task this "architecture" is a nightmare waiting to happen.
- robohamburger 9y ago
  Fair enough. It seems like it still ought to be able to burn the kafka+elasticsearch world down and submit everything to kafka with such a setup (and thus elasticsearch). I would certainly not sleep very well at night if I could not.
miguno 9y ago
> I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically). That (replaying if needed) is exactly what Kafka allows you to do, unless I misunderstood what you wrote.
- robohamburger 9y ago

_lex 9y ago

I think the POV he's taken is that the kafka stream is the one true datasource (for all time), with all other dbs being derivatives. This insane strategy seems to be over engineered to get around db migrations... though i'm sure the event stream will also change over time, and he'll have to write migration-like code anyways.

rhizome 9y ago
If datasource engineers are for some reason implementing "ontogeny recapitulates phylogeny," they won't be creating migration-like code, they'll be writing ETL filters. Break out your Members Only jackets!
LgWoodenBadger 9y ago
There's not just one reason for it. Kafka lets dependent consumers transform that data into whatever model is appropriate for their use case. There's no one-size-fits-all E-R model for data for all use cases. It's not just working around db migrations. It's also providing you with the ability to model the data as many ways as you require. How else would you do it, materialized views? How often are they materialized? Regular views? How performant are they? There are many benefits to this approach, and a lot of them require a different way of thinking. It's a different paradigm.
- dreamfactored 9y ago
  Isn't that just skinny model, fat client?