13 ms·

It’s Okay to Store Data in Apache Kafka (2017)

84 points by ooooak 7y ago

vore 7y ago
I think what the article greatly skimps over is data migrations: what do you do if you need to change the format of your data? If you retain logs in Kafka indefinitely as the source of truth for your data, then if you need to migrate materialized data to a new format, you'll also need to either 1) support all the previous forms of materialized data so operations from the log are guaranteed to be safely replayable on it, or 2) don't do that and keep one form of materialized data and hope you have enough test coverage to make sure some unexpectedly old data doesn't silently corrupt your materialized data. Event sourcing is useful, but using it as a source of truth data store in itself instead of e.g. an occasional journalling mechanism seems pretty fraught.
- Joeri 7y ago
  You use AVRO and the confluent schema store so that you can do compatible schema upgrades (automatically upgrading messages when reading). If you need to do a breaking change, you can detect the avro schema from the message and have code in the reader to convert one to the other. It’s not that much of a problem in practice.
  - vore 7y ago
    This is a problem we've actually encountered in practice: you need to preserve the conversion code for as long as the history exists, which is permanent technical debt. If you get your domain model wrong at the start, it will haunt you forever.
    - parasubvert 7y ago
      This is no different from evolving an RDBMS with ActiveRecord or Flyway. It’s not technical debt (there is nothing to repay or maintain).
      - dllthomas 7y ago
        If you write "read it to the latest version" code for every version, you need to write that for every past version - potentially a high burden. Writing "convert it to the next version" instead is much less work for humans, and probably usually the best call, but less efficient when you actually need to read that old data. There are hybrid approaches to consider, too. Wherever you are on that spectrum, there is code. Exercising it may slow down your test suite. It might need to be touched when you update lint rules (at least to add annotations to shut off the new rules). Code nearly always represents some amount of debt. Code to deal with history, more so.
        parasubvert 7y ago
        We have a slight disagreement on what it means to accrue technical debt. When a schema changes for an all, someone has to migrate it and the data, unless the design is that all incompatible data is dropped. Either that’s the codebase itself or the dev team is punting to the user. Punting to the user just shifts the burden around. If the R&D team shipped a shitty / incomplete schema to get the software out the door that they need to change later, then yes, that is technical debt - something they’ll eventually need to repay. If requirements evolve over time and thus the schema needs to, that is not necessarily technical debt in the usual sense, which usually implies a temporary technical compromise for the purposes of expediency / getting something out the door. I suppose I agree there is a trade off here and a penalty - after many years the migrations get slow to apply etc, and you could say that checkpointing the schema every few versions and preventing upgrades from any prior point is a way of cleaning up the “migration debt”. But people like to suggest that there’s some other way , ie. the OP saying “ If you get your domain model wrong at the start, it will haunt you forever.”.... I have never seen any system get the domain model perfectly right at the start!
        dllthomas 7y ago
        That may be. I was mostly trying to address the claim that "[i]t's not technical debt [because] there is nothing to [...] maintain." Financially speaking, it's more like an annuity than a credit card.
    - Fiahil 7y ago
      You can truncate the history by reading events from the starting point to the junction, then creating a new snapshot that will serve as a new starting point, and finally deleting the previous history segment.
    - dwenzek 7y ago
      This is definitely a decision which requires a long term support, as any solution regarding storage. This is not a technical debt unless you see it as a temporary solution, waiting for a better time to replace it. You highlight the "need to preserve the conversion code for as long as the history exists". This is a point so decisive that I even wonder if Kafka should provide some support to keep that relationship.
    - lojack 7y ago
      So, like a set of traditional database migrations?
  - tele_ski 7y ago
    I personally don't understand avro, are there tools to write the schemas easier? The json format is very difficult to read and it just seems so clunky compared to protobuf.
- abalone 7y ago
  > 1) support all the previous forms of materialized data so operations from the log are guaranteed to be safely replayable on it Yes, I believe with event sourcing you typically do exactly that. The very point of it is to use the event log as the source of truth, not just an “occasional journaling mechanism.”
  - mumblemumble 7y ago
    I've worked on maintaining systems like that. IME, it's classic technical debt: You can get up and running much more quickly and cheaply, but in the long run it can become quite expensive. So, like any good technical debt, it's fine, even desirable, in the short run, but less fine if you end up carrying it indefinitely. Fortunately, Kafka's already thought about this, so, when the time comes, it should be pretty easy to migrate to an architecture where you're no longer using it as your long-term source of truth.
    - abalone 7y ago
      Event sourcing may be many things but I don't think "quick and cheap" is one of them. If you want quick and cheap do a (distributed) monolith.
    - UK-Al05 7y ago
      People go out there way to make events the source of truth in non-kafka systems. With a lot of effort involved. It isn't just people being lazy with Kafka. Event sourcing existed before with traditional databases. I came across it in the .net world with cqrs + event sourcing combo long before Kafka became popular. People put a lot effort in to move away traditional current state storage. Often used in industries that have a lot of regulation.
- UK-Al05 7y ago
  The entire point of event sourcing is that it is the source of truth. The sourcing in event sourcing. Otherwise it's not different from a audit log.
  - zbentley 7y ago
    I think what GP meant is: what if one of your "indexes" is a SQL database, and over time the shape that database needs to be changes? In that situation, how do you handle migrations on that database, or building a new copy of it from scratch via the log? Your log will have historical data in a different shape than the schema expects, so you'll end up in an uncomfortable "replay a little, migrate, repeat" situation when reading historical data into the index for any reason.
    - UK-Al05 7y ago
      Normally you version your events, uplift or convert a stream to a updated version.
KaiserPro 7y ago
you _can_ do it, but you shouldn't treating your message bus as a infinite storage system is going give you a bad time.
- marton78 7y ago
  This is quite an unsubstantiated claim in response to a detailed article. Care to elaborate?
  - gopalv 7y ago
    Not the original comment, but I have run into a specific set of regulatory scenarios where the immutability of any event log is a big problem. The data retention policies for users who leave the platform is set to 30 days by GDPR, which requires the data to be deleted and expunged by the storage system in ways in which normal users cannot recover - or in another way, the data needs to "offline". This is not actually that "all data older than 30 days is thrown away", but that "all data for users who have deleted their accounts need to be deleted from the start of their account creation, 30 days after they say forget-me". Kafka (or Pulsar or Pravega) or any of the other immutable commit log implementations make forgetting a small slice of data from a large set a complicated and nearly impossible task to accomplish. You can accomplish some part of this with log compaction assuming you have a definite primary key for all updates (i.e the log needs to be partitioned on a key to do compaction along that key). If there's a way I could declare a primary key as a device+event_timestamp+metric, but delete by a user_id column, let me know and I'll be happy to find out how to do it. However in its original form, Kafka is still very useful. Being able to hold data in Kafka in those periods of time is extremely valuable and naturally lets the system lose part of its state with the ability to replay itself back into the same state from a saved checkpoint. If you store 7+ days of Kafka data and flush the newly arrived data-set into a persistent, but mutable columnar store every day & maintain the partition/offsets on commit, then you can recover from a complete loss of the mutable store's in-memory data by replaying the log from where you left off. The row-major nature of its storage still hurts though if you plan to do all your analysis off it directly, because you'll burn through the disk bandwidth for no good reason.
epistasis 7y ago
Kreps has a gift for writing, this is so clear, well organized, and far more fun to read than the topic has any right to be. Hopefully he'll retire after confluent and finally start writing novels.
- sitkack 7y ago
  I came to say something similar, but targeted at the quoted paragraph below. > If Kafka isn’t going to be a universal format for queries, what is it? I think it makes more sense to think of your datacenter as a giant database, in that database Kafka is the commit log, and these various storage systems are kinds of derived indexes or views. It’s true that a log like Kafka can be fantastic primitive for building a database, but the queries are still served out of indexes that were built for the appropriate access pattern. A wonderful insight!
ryanthedev 7y ago
Well maybe for non critical data. Multi regional Kafka clustering is not easy. There are much better and cheaper data storage options that can provide eventual consistency.
- je42 7y ago
  What tech would you use to store a critical append-only data if not with Kafka ?
  - super_mario 7y ago
    Appache Cassandra.
    - je42 7y ago
      ok,in what areas is Cassandra better than Kafka ? i am reading this https://stackoverflow.com/questions/35711129/how-to-stream-data-out-of-a-cassandra-table https://stackoverflow.com/questions/35711129/how-to-stream-d... According to https://issues.apache.org/jira/browse/CASSANDRA-8844 https://issues.apache.org/jira/browse/CASSANDRA-8844: CDC seems a lot less capable than Kafka: > Cassandra would only write to the CDC log, and never delete from it. > Cleaning up consumed logfiles would be the client daemon's responibility on the other hand there is the Cassandra CDC Kafka Connector, that seems to do some work... in the i still don't see why i would want to use Cassandra over Kafka for storing data, when i care about durability and streaming of append only data.
  - rockostrich 7y ago
    Kafka topics compacted to some sort of disk storage? It really depends on what you need to do with it. All of our kafka topics have a TTL on messages that's less than or equal to 7 days. If the data is critical, then that topic has a archiver job that writes the messages to files in HDFS. Postgres, Spanner, etc. come to mind as options if you need to store critical append-only data and also have it be queryable in real-time.
    - 7y ago
mvitorino 7y ago
IMO using Kafka for long term storage is not the greatest idea. It is expensive to keep CPU and RAM constantly on top of data that it going to be cold most of the time. There is no DML which means mistakes are expensive (from an engineering pov). And while the whole event sourcing paradigm can work quite well in narrow domains with teams fully aware of the implications of what they are doing, in practice, on large orgs, it is hard to scale (from a people perspective).
- tempodox 7y ago
  > hard to scale (from a people perspective). This is always going to be true. Either your system is built in a way that even the most ignorant of users cannot do permanent damage, or you design it to be understood and run by a small group of people who fully comprehend what they're doing. Of course neither option is easy, but the alternative is a prolific generator of daily trouble.
zbentley 7y ago
This article seems to propose log compaction as the answer to the question of size (i.e. how much historical data is going to have to be kept around and how much is that going to cost). However, log compaction is not well suited to many use cases: storing partial updates or diffs in the log, storing many trillions of tiny entries (as keys), multiple messages on the log corresponding to related (contingent) updates, and so on. Those are tractable but hard to solve; log compaction is not a silver bullet and unless you think really hard about how your data changes over time, you may end up storing more of it than you expect if you use the log as an eternal source of truth--compaction or not.

84 points by ooooak 7y ago

vore 7y ago

I think what the article greatly skimps over is data migrations: what do you do if you need to change the format of your data? If you retain logs in Kafka indefinitely as the source of truth for your data, then if you need to migrate materialized data to a new format, you'll also need to either 1) support all the previous forms of materialized data so operations from the log are guaranteed to be safely replayable on it, or 2) don't do that and keep one form of materialized data and hope you have enough test coverage to make sure some unexpectedly old data doesn't silently corrupt your materialized data. Event sourcing is useful, but using it as a source of truth data store in itself instead of e.g. an occasional journalling mechanism seems pretty fraught.

Joeri 7y ago
You use AVRO and the confluent schema store so that you can do compatible schema upgrades (automatically upgrading messages when reading). If you need to do a breaking change, you can detect the avro schema from the message and have code in the reader to convert one to the other. It’s not that much of a problem in practice.
- vore 7y ago
  This is a problem we've actually encountered in practice: you need to preserve the conversion code for as long as the history exists, which is permanent technical debt. If you get your domain model wrong at the start, it will haunt you forever.
  - parasubvert 7y ago
    This is no different from evolving an RDBMS with ActiveRecord or Flyway. It’s not technical debt (there is nothing to repay or maintain).
    - dllthomas 7y ago
      If you write "read it to the latest version" code for every version, you need to write that for every past version - potentially a high burden. Writing "convert it to the next version" instead is much less work for humans, and probably usually the best call, but less efficient when you actually need to read that old data. There are hybrid approaches to consider, too. Wherever you are on that spectrum, there is code. Exercising it may slow down your test suite. It might need to be touched when you update lint rules (at least to add annotations to shut off the new rules). Code nearly always represents some amount of debt. Code to deal with history, more so.
      - parasubvert 7y ago
        We have a slight disagreement on what it means to accrue technical debt. When a schema changes for an all, someone has to migrate it and the data, unless the design is that all incompatible data is dropped. Either that’s the codebase itself or the dev team is punting to the user. Punting to the user just shifts the burden around. If the R&D team shipped a shitty / incomplete schema to get the software out the door that they need to change later, then yes, that is technical debt - something they’ll eventually need to repay. If requirements evolve over time and thus the schema needs to, that is not necessarily technical debt in the usual sense, which usually implies a temporary technical compromise for the purposes of expediency / getting something out the door. I suppose I agree there is a trade off here and a penalty - after many years the migrations get slow to apply etc, and you could say that checkpointing the schema every few versions and preventing upgrades from any prior point is a way of cleaning up the “migration debt”. But people like to suggest that there’s some other way , ie. the OP saying “ If you get your domain model wrong at the start, it will haunt you forever.”.... I have never seen any system get the domain model perfectly right at the start!
        dllthomas 7y ago
        That may be. I was mostly trying to address the claim that "[i]t's not technical debt [because] there is nothing to [...] maintain." Financially speaking, it's more like an annuity than a credit card.
  - Fiahil 7y ago
    You can truncate the history by reading events from the starting point to the junction, then creating a new snapshot that will serve as a new starting point, and finally deleting the previous history segment.
  - dwenzek 7y ago
    This is definitely a decision which requires a long term support, as any solution regarding storage. This is not a technical debt unless you see it as a temporary solution, waiting for a better time to replace it. You highlight the "need to preserve the conversion code for as long as the history exists". This is a point so decisive that I even wonder if Kafka should provide some support to keep that relationship.
  - lojack 7y ago
    So, like a set of traditional database migrations?
- tele_ski 7y ago
  I personally don't understand avro, are there tools to write the schemas easier? The json format is very difficult to read and it just seems so clunky compared to protobuf.
abalone 7y ago
> 1) support all the previous forms of materialized data so operations from the log are guaranteed to be safely replayable on it Yes, I believe with event sourcing you typically do exactly that. The very point of it is to use the event log as the source of truth, not just an “occasional journaling mechanism.”
- mumblemumble 7y ago
  I've worked on maintaining systems like that. IME, it's classic technical debt: You can get up and running much more quickly and cheaply, but in the long run it can become quite expensive. So, like any good technical debt, it's fine, even desirable, in the short run, but less fine if you end up carrying it indefinitely. Fortunately, Kafka's already thought about this, so, when the time comes, it should be pretty easy to migrate to an architecture where you're no longer using it as your long-term source of truth.
  - abalone 7y ago
    Event sourcing may be many things but I don't think "quick and cheap" is one of them. If you want quick and cheap do a (distributed) monolith.
  - UK-Al05 7y ago
    People go out there way to make events the source of truth in non-kafka systems. With a lot of effort involved. It isn't just people being lazy with Kafka. Event sourcing existed before with traditional databases. I came across it in the .net world with cqrs + event sourcing combo long before Kafka became popular. People put a lot effort in to move away traditional current state storage. Often used in industries that have a lot of regulation.
UK-Al05 7y ago
The entire point of event sourcing is that it is the source of truth. The sourcing in event sourcing. Otherwise it's not different from a audit log.
- zbentley 7y ago
  I think what GP meant is: what if one of your "indexes" is a SQL database, and over time the shape that database needs to be changes? In that situation, how do you handle migrations on that database, or building a new copy of it from scratch via the log? Your log will have historical data in a different shape than the schema expects, so you'll end up in an uncomfortable "replay a little, migrate, repeat" situation when reading historical data into the index for any reason.
  - UK-Al05 7y ago
    Normally you version your events, uplift or convert a stream to a updated version.

KaiserPro 7y ago

you _can_ do it, but you shouldn't treating your message bus as a infinite storage system is going give you a bad time.

marton78 7y ago
This is quite an unsubstantiated claim in response to a detailed article. Care to elaborate?
- gopalv 7y ago
  Not the original comment, but I have run into a specific set of regulatory scenarios where the immutability of any event log is a big problem. The data retention policies for users who leave the platform is set to 30 days by GDPR, which requires the data to be deleted and expunged by the storage system in ways in which normal users cannot recover - or in another way, the data needs to "offline". This is not actually that "all data older than 30 days is thrown away", but that "all data for users who have deleted their accounts need to be deleted from the start of their account creation, 30 days after they say forget-me". Kafka (or Pulsar or Pravega) or any of the other immutable commit log implementations make forgetting a small slice of data from a large set a complicated and nearly impossible task to accomplish. You can accomplish some part of this with log compaction assuming you have a definite primary key for all updates (i.e the log needs to be partitioned on a key to do compaction along that key). If there's a way I could declare a primary key as a device+event_timestamp+metric, but delete by a user_id column, let me know and I'll be happy to find out how to do it. However in its original form, Kafka is still very useful. Being able to hold data in Kafka in those periods of time is extremely valuable and naturally lets the system lose part of its state with the ability to replay itself back into the same state from a saved checkpoint. If you store 7+ days of Kafka data and flush the newly arrived data-set into a persistent, but mutable columnar store every day & maintain the partition/offsets on commit, then you can recover from a complete loss of the mutable store's in-memory data by replaying the log from where you left off. The row-major nature of its storage still hurts though if you plan to do all your analysis off it directly, because you'll burn through the disk bandwidth for no good reason.

epistasis 7y ago

Kreps has a gift for writing, this is so clear, well organized, and far more fun to read than the topic has any right to be. Hopefully he'll retire after confluent and finally start writing novels.

sitkack 7y ago
I came to say something similar, but targeted at the quoted paragraph below. > If Kafka isn’t going to be a universal format for queries, what is it? I think it makes more sense to think of your datacenter as a giant database, in that database Kafka is the commit log, and these various storage systems are kinds of derived indexes or views. It’s true that a log like Kafka can be fantastic primitive for building a database, but the queries are still served out of indexes that were built for the appropriate access pattern. A wonderful insight!

ryanthedev 7y ago

Well maybe for non critical data. Multi regional Kafka clustering is not easy. There are much better and cheaper data storage options that can provide eventual consistency.

je42 7y ago
What tech would you use to store a critical append-only data if not with Kafka ?
- super_mario 7y ago
  Appache Cassandra.
  - je42 7y ago
    ok,in what areas is Cassandra better than Kafka ? i am reading this https://stackoverflow.com/questions/35711129/how-to-stream-data-out-of-a-cassandra-table https://stackoverflow.com/questions/35711129/how-to-stream-d... According to https://issues.apache.org/jira/browse/CASSANDRA-8844 https://issues.apache.org/jira/browse/CASSANDRA-8844: CDC seems a lot less capable than Kafka: > Cassandra would only write to the CDC log, and never delete from it. > Cleaning up consumed logfiles would be the client daemon's responibility on the other hand there is the Cassandra CDC Kafka Connector, that seems to do some work... in the i still don't see why i would want to use Cassandra over Kafka for storing data, when i care about durability and streaming of append only data.
- rockostrich 7y ago
  Kafka topics compacted to some sort of disk storage? It really depends on what you need to do with it. All of our kafka topics have a TTL on messages that's less than or equal to 7 days. If the data is critical, then that topic has a archiver job that writes the messages to files in HDFS. Postgres, Spanner, etc. come to mind as options if you need to store critical append-only data and also have it be queryable in real-time.
  - 7y ago

mvitorino 7y ago

IMO using Kafka for long term storage is not the greatest idea. It is expensive to keep CPU and RAM constantly on top of data that it going to be cold most of the time. There is no DML which means mistakes are expensive (from an engineering pov). And while the whole event sourcing paradigm can work quite well in narrow domains with teams fully aware of the implications of what they are doing, in practice, on large orgs, it is hard to scale (from a people perspective).

tempodox 7y ago
> hard to scale (from a people perspective). This is always going to be true. Either your system is built in a way that even the most ignorant of users cannot do permanent damage, or you design it to be understood and run by a small group of people who fully comprehend what they're doing. Of course neither option is easy, but the alternative is a prolific generator of daily trouble.

zbentley 7y ago

This article seems to propose log compaction as the answer to the question of size (i.e. how much historical data is going to have to be kept around and how much is that going to cost). However, log compaction is not well suited to many use cases: storing partial updates or diffs in the log, storing many trillions of tiny entries (as keys), multiple messages on the log corresponding to related (contingent) updates, and so on. Those are tractable but hard to solve; log compaction is not a silver bullet and unless you think really hard about how your data changes over time, you may end up storing more of it than you expect if you use the log as an eternal source of truth--compaction or not.