3 ms·

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements a

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale. I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

qaq 9y ago
They are storing 100GB :). You can do ok with 5TB +/- on something as crappy as RDS.
pcsanwald 9y ago
The author states it's about 100GB of data.
- TwelveNights 9y ago
  Is 100GB of data really enough to warrant this type of architecture?
  - elnygren 9y ago
    if you read more closely, this article is not about the amount of data, but the way how it is managed.
    - tlberglund 9y ago
      Precisely. While "all NYT data since 1851" sounds like a lot, and >8660 days sounds like a long retention period for a Kafka topic, this--like most systems in the world--is not a Big Data application. One of the key insights from the post is that there are interesting architectural considerations that have nothing to do with data size that make immutable logs a good idea.
theossuary 9y ago
See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions. Also I've never thought of Kafka as a persistent data storage solution, it's interesting Confluent is supporting Kafka being used in this way.
- tlberglund 9y ago
  You are not even a little bit confused! I think you have it perfectly right. And you are not alone in not thinking of Kafka as persistent storage, but when you get down to it there is no reason not to, and people are indeed using it in just that way. And yes, Confluent does give its +1 to this practice. :)
- poooogles 9y ago
  >it's interesting Confluent is supporting Kafka being used in this way. If it earns them money, I think they'll pretty much support anything. Jay has rubber stamped it on SO [1], but he's got a bit of a vested interest on selling Kafka. 1. https://stackoverflow.com/questions/17708489/using-kafka-as-a-cqrs-eventstore-good-idea/22597637#22597637 https://stackoverflow.com/questions/17708489/using-kafka-as-...
deleted 9y ago
[deleted]
pram 9y ago
Kafka by default is not persistent, the logs expire after 7 days. You can increase it on a per topic basis. It also doesn't scale automatically. If you have three replicas on a single partition topic, they will live on their assigned broker forever unless you manually reassign them. Adding new nodes does not kick off rebalancing of partitions. Its automatic cluster management is very primitive compared to something like elasticsearch. For example, if you lose a broker, the replica will just be gone forever. Unless you replace the broker with the same id.
- tlberglund 9y ago
  Your statements about data retention are true, but if I may, I think they are only vacuously true. There are definitely ways to configure data retention incorrectly, but it seems like the folks at NYT have cracked the nut. Indeed, this is a trivial thing to set up. Your points about auto-scaling are well-taken, though. It does not automatically rebalance when you add a new node. Confluent obvious agrees that this is a problem, since this is part of what Confluent Enterprise does. #shamelessplug (Which of course I do not intend shamelessly; just saying you have a point. :) )

qaq 9y ago

They are storing 100GB :). You can do ok with 5TB +/- on something as crappy as RDS.

pcsanwald 9y ago

The author states it's about 100GB of data.

TwelveNights 9y ago
Is 100GB of data really enough to warrant this type of architecture?
- elnygren 9y ago
  if you read more closely, this article is not about the amount of data, but the way how it is managed.
  - tlberglund 9y ago
    Precisely. While "all NYT data since 1851" sounds like a lot, and >8660 days sounds like a long retention period for a Kafka topic, this--like most systems in the world--is not a Big Data application. One of the key insights from the post is that there are interesting architectural considerations that have nothing to do with data size that make immutable logs a good idea.

theossuary 9y ago

See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions. Also I've never thought of Kafka as a persistent data storage solution, it's interesting Confluent is supporting Kafka being used in this way.

tlberglund 9y ago
You are not even a little bit confused! I think you have it perfectly right. And you are not alone in not thinking of Kafka as persistent storage, but when you get down to it there is no reason not to, and people are indeed using it in just that way. And yes, Confluent does give its +1 to this practice. :)
poooogles 9y ago
>it's interesting Confluent is supporting Kafka being used in this way. If it earns them money, I think they'll pretty much support anything. Jay has rubber stamped it on SO [1], but he's got a bit of a vested interest on selling Kafka. 1. https://stackoverflow.com/questions/17708489/using-kafka-as-a-cqrs-eventstore-good-idea/22597637#22597637 https://stackoverflow.com/questions/17708489/using-kafka-as-...

deleted 9y ago

[deleted]

pram 9y ago

Kafka by default is not persistent, the logs expire after 7 days. You can increase it on a per topic basis. It also doesn't scale automatically. If you have three replicas on a single partition topic, they will live on their assigned broker forever unless you manually reassign them. Adding new nodes does not kick off rebalancing of partitions. Its automatic cluster management is very primitive compared to something like elasticsearch. For example, if you lose a broker, the replica will just be gone forever. Unless you replace the broker with the same id.

tlberglund 9y ago
Your statements about data retention are true, but if I may, I think they are only vacuously true. There are definitely ways to configure data retention incorrectly, but it seems like the folks at NYT have cracked the nut. Indeed, this is a trivial thing to set up. Your points about auto-scaling are well-taken, though. It does not automatically rebalance when you add a new node. Confluent obvious agrees that this is a problem, since this is part of what Confluent Enterprise does. #shamelessplug (Which of course I do not intend shamelessly; just saying you have a point. :) )