6 ms·

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the sc

by toomim 9y ago

> Traditionally, databases have been used as the source of truth ... [but] can be difficult to manage in the long run. First, it’s often tricky to change the schema of a database. Adding and removing fields is not too hard, but more fundamental schema changes can be difficult to organize without downtime. This argument sounds self-contradicting. Kafka doesn't let you change its schema at all! At least postgres gives you the option. It seems that the author is excited about having a single source of truth that doesn't change, and didn't realize that he could do that with a database, if he just never used the schema-changing features. Am I missing something? It seems like the author could be totally happy with a bunch of derived postgres databases sitting in front of a "source of truth" database, where he never changes the source of truth database's schema. Why use kafka?

elnygren 9y ago
Postgres could in fact be used here by creating an append-only table similar to: id, data (JSON field) However, Postgres doesn't have good support for creating derived append-only logs (=streams) from that table. Kafka has Kafka Streams and KafkaSQL. And producer+consumer APIs that are a good fit for NYTs use case. > Am I missing something? Remember that in addition to schema changes, NYT also wants to avoid row changes. One reason was that all search indices & other systems need updating too during a row change in a DB and this will lead to inconsistencies in large scale (sometimes some of these updates fail). IMHO the thing you are missing is the log based architecture where all databases are derived/materialised from the SSOT: the log.
- qaq 9y ago
  However, Postgres doesn't have good support for creating derived append-only logs ???? A one line trigger will give you derived append only log that is transactionally consistent.
  - elnygren 9y ago
    so how does that one-liner transform data from log A to log B in realtime? are talking about creating a new table or materialised views or stored procedures or...?
    - qaq 9y ago
      In your post you are proposing to have a table be equivalent to a kafka topic. So derived "log"/topic can be another table updated with a trigger.
- manigandham 9y ago
  Pretty much all databases are already a transaction log that gets materialized. Postgres and many other RDBMS take in changes, write to the WAL, then provide tables that are views of the latest data of each row. If you want total history and the database doesn't support this automatically then you can easily insert news rows instead (like you described) and then just create an SQL view that then shows the latest versions of each row, while also adding views for all kinds of other data access patterns. Companies have been doing this for decades because it's self-contained, fast, and reliable. Using Kafka and separate processes to do this is deconstructing the RDBMS into separate layers that you now have to manage yourself. Useful if you really have that kind of scale but at 100GB of data, it's just silly. Use kafka as a work queue but leave the database work to actual database software.
BenoitEssiambre 9y ago
I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.
- qaq 9y ago
  They are storing 100GB :). You can do ok with 5TB +/- on something as crappy as RDS.
- pcsanwald 9y ago
  The author states it's about 100GB of data.
  - TwelveNights 9y ago
    Is 100GB of data really enough to warrant this type of architecture?
    - elnygren 9y ago
      if you read more closely, this article is not about the amount of data, but the way how it is managed.
      - tlberglund 9y ago
        Precisely. While "all NYT data since 1851" sounds like a lot, and >8660 days sounds like a long retention period for a Kafka topic, this--like most systems in the world--is not a Big Data application. One of the key insights from the post is that there are interesting architectural considerations that have nothing to do with data size that make immutable logs a good idea.
- theossuary 9y ago
  See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions. Also I've never thought of Kafka as a persistent data storage solution, it's interesting Confluent is supporting Kafka being used in this way.
sidlls 9y ago
I agree. It's rare that a company needs something like Kafka. Kafka introduces a number of issues related to the development of client code and data stores (if any) and the maintenance of these things. It's important that the actual scale justifies the expense incurred.
- qaq 9y ago
  It's not rare, using Kafka in place of a prober db is crazy though. There are tones of use cases that benefit from having Kafka a very common one is putting Kafka in front of the data processing pipeline to absorb spikes without data loss.
  - sidlls 9y ago
    It's rare that a company has spikes of that sort that can't be handled in far more mundane ways. Most companies do not operate at a scale that merits this sort of solution. Look at the engineering blogs on LinkedIn for the reasoning behind creating it. It's over-kill for most use cases.
    - qaq 9y ago
      Kafka is fairly mundane piece of software aside from periodic ops issues with Zookeeper.
robertlagrant 9y ago
Heh this was exactly my thought. I've no idea why they couldn't just store the data in a database! Or if they really have several ways of storing data, then have a database for each and a common API/ESB layer over the top. Seems normal.
kod 9y ago
Kafka doesn't have a schema per message, messages are just bytes that you can serialize / deserialize however you want to. The article refers to using protobuf for messages, which does support adding fields. If you're equating kafka topics with the idea of a schema, you can add topics.
- l8again 9y ago
  To add to this. Having implemented a similar log-based architecture, I would say that it is much simpler data infrastructure than having a central RDBMS considering the use case. Remember we need to deal with several consumer applications with their own respective optimization models. Postgres is a perfect choice for a given application's local datastore, while for a different application they may want to use ElasticSearch as their datastore. However, the "source of truth" remains free of any such optimization requirements. You simply save your "messages", "event", "facts", whatever you want to call it in its purest form preferably immutable, and let the consumer apps create/recreate their local datastores as they deem fit.
monktastic1 9y ago
Postgres doesn't help with the streaming part though, does it?
lobster_johnson 9y ago
The problem with an RDBMS is that the schema is retroactive -- every time you update the schema, both old and new data must validate against it. There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable. The most trivial benefit is that no data is ever lost, even if the business case is taken away. Over time, the burden of supporting obsolete features mean that you want to remove columns that don't apply anymore. But that does remove interesting and potentially valuable historical data. If the schema evolves together with the data, you can keep even the cruft, at little expense. In practice, the evolution of the schema means that data often has to be transformed from one schema to another (e.g. v1 -> v2). In an RDBMS, this transformation (schema migration) is performed once, and if it contains bugs, you have to resort to backups to restore the old data. But with a versioned schema, the transformations are simply functions on immutable data. You can go back and re-run transformations on old data to get new data, non-destructively.
- turbohedgehog 9y ago
  I think this is what Valve does with Dota 2 replays by using protobuf
- seanp2k2 9y ago
  CouchDB is cool if you want to mess with revisions: http://docs.couchdb.org/en/2.1.0/intro/api.html http://docs.couchdb.org/en/2.1.0/intro/api.html When you're talking billions - trillions of events, check out something like https://www.slideshare.net/AmazonWebServices/how-netflix-uses-amazon-kinesis-streams-to-monitor-and-optimize-largescale-networks-in-realtime https://www.slideshare.net/AmazonWebServices/how-netflix-use...
- qaq 9y ago
  id uuid ts timestsmp data jsonb solved it for you :) and thats will have ACID and all the other features of RDBMS that kafka does not have. It will also allow you to create transactionally consistent "views" representing whatever you need implemented in a few lines of SQL vs complex standalone services.

by toomim 9y ago

elnygren 9y ago

Postgres could in fact be used here by creating an append-only table similar to: id, data (JSON field) However, Postgres doesn't have good support for creating derived append-only logs (=streams) from that table. Kafka has Kafka Streams and KafkaSQL. And producer+consumer APIs that are a good fit for NYTs use case. > Am I missing something? Remember that in addition to schema changes, NYT also wants to avoid row changes. One reason was that all search indices & other systems need updating too during a row change in a DB and this will lead to inconsistencies in large scale (sometimes some of these updates fail). IMHO the thing you are missing is the log based architecture where all databases are derived/materialised from the SSOT: the log.

qaq 9y ago
However, Postgres doesn't have good support for creating derived append-only logs ???? A one line trigger will give you derived append only log that is transactionally consistent.
- elnygren 9y ago
  so how does that one-liner transform data from log A to log B in realtime? are talking about creating a new table or materialised views or stored procedures or...?
  - qaq 9y ago
    In your post you are proposing to have a table be equivalent to a kafka topic. So derived "log"/topic can be another table updated with a trigger.
manigandham 9y ago
Pretty much all databases are already a transaction log that gets materialized. Postgres and many other RDBMS take in changes, write to the WAL, then provide tables that are views of the latest data of each row. If you want total history and the database doesn't support this automatically then you can easily insert news rows instead (like you described) and then just create an SQL view that then shows the latest versions of each row, while also adding views for all kinds of other data access patterns. Companies have been doing this for decades because it's self-contained, fast, and reliable. Using Kafka and separate processes to do this is deconstructing the RDBMS into separate layers that you now have to manage yourself. Useful if you really have that kind of scale but at 100GB of data, it's just silly. Use kafka as a work queue but leave the database work to actual database software.

BenoitEssiambre 9y ago

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

qaq 9y ago
They are storing 100GB :). You can do ok with 5TB +/- on something as crappy as RDS.
pcsanwald 9y ago
The author states it's about 100GB of data.
- TwelveNights 9y ago
  Is 100GB of data really enough to warrant this type of architecture?
  - elnygren 9y ago
    if you read more closely, this article is not about the amount of data, but the way how it is managed.
    - tlberglund 9y ago
      Precisely. While "all NYT data since 1851" sounds like a lot, and >8660 days sounds like a long retention period for a Kafka topic, this--like most systems in the world--is not a Big Data application. One of the key insights from the post is that there are interesting architectural considerations that have nothing to do with data size that make immutable logs a good idea.
theossuary 9y ago
See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions. Also I've never thought of Kafka as a persistent data storage solution, it's interesting Confluent is supporting Kafka being used in this way.

sidlls 9y ago

I agree. It's rare that a company needs something like Kafka. Kafka introduces a number of issues related to the development of client code and data stores (if any) and the maintenance of these things. It's important that the actual scale justifies the expense incurred.

qaq 9y ago
It's not rare, using Kafka in place of a prober db is crazy though. There are tones of use cases that benefit from having Kafka a very common one is putting Kafka in front of the data processing pipeline to absorb spikes without data loss.
- sidlls 9y ago
  It's rare that a company has spikes of that sort that can't be handled in far more mundane ways. Most companies do not operate at a scale that merits this sort of solution. Look at the engineering blogs on LinkedIn for the reasoning behind creating it. It's over-kill for most use cases.
  - qaq 9y ago
    Kafka is fairly mundane piece of software aside from periodic ops issues with Zookeeper.

robertlagrant 9y ago

Heh this was exactly my thought. I've no idea why they couldn't just store the data in a database! Or if they really have several ways of storing data, then have a database for each and a common API/ESB layer over the top. Seems normal.

kod 9y ago

Kafka doesn't have a schema per message, messages are just bytes that you can serialize / deserialize however you want to. The article refers to using protobuf for messages, which does support adding fields. If you're equating kafka topics with the idea of a schema, you can add topics.

l8again 9y ago
To add to this. Having implemented a similar log-based architecture, I would say that it is much simpler data infrastructure than having a central RDBMS considering the use case. Remember we need to deal with several consumer applications with their own respective optimization models. Postgres is a perfect choice for a given application's local datastore, while for a different application they may want to use ElasticSearch as their datastore. However, the "source of truth" remains free of any such optimization requirements. You simply save your "messages", "event", "facts", whatever you want to call it in its purest form preferably immutable, and let the consumer apps create/recreate their local datastores as they deem fit.

monktastic1 9y ago

Postgres doesn't help with the streaming part though, does it?

lobster_johnson 9y ago

The problem with an RDBMS is that the schema is retroactive -- every time you update the schema, both old and new data must validate against it. There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable. The most trivial benefit is that no data is ever lost, even if the business case is taken away. Over time, the burden of supporting obsolete features mean that you want to remove columns that don't apply anymore. But that does remove interesting and potentially valuable historical data. If the schema evolves together with the data, you can keep even the cruft, at little expense. In practice, the evolution of the schema means that data often has to be transformed from one schema to another (e.g. v1 -> v2). In an RDBMS, this transformation (schema migration) is performed once, and if it contains bugs, you have to resort to backups to restore the old data. But with a versioned schema, the transformations are simply functions on immutable data. You can go back and re-run transformations on old data to get new data, non-destructively.

turbohedgehog 9y ago
I think this is what Valve does with Dota 2 replays by using protobuf
seanp2k2 9y ago
CouchDB is cool if you want to mess with revisions: http://docs.couchdb.org/en/2.1.0/intro/api.html http://docs.couchdb.org/en/2.1.0/intro/api.html When you're talking billions - trillions of events, check out something like https://www.slideshare.net/AmazonWebServices/how-netflix-uses-amazon-kinesis-streams-to-monitor-and-optimize-largescale-networks-in-realtime https://www.slideshare.net/AmazonWebServices/how-netflix-use...
qaq 9y ago
id uuid ts timestsmp data jsonb solved it for you :) and thats will have ACID and all the other features of RDBMS that kafka does not have. It will also allow you to create transactionally consistent "views" representing whatever you need implemented in a few lines of SQL vs complex standalone services.