4 ms·

> this exact mistake https://kafka-summit.org/sessions/source-truth-new-york-times-stores-every-piece-content-ever-published-kafka/ https://kafka-summit.org/se

> this exact mistake https://kafka-summit.org/sessions/source-truth-new-york-times-stores-every-piece-content-ever-published-kafka/ https://kafka-summit.org/sessions/source-truth-new-york-time... Both your comment and the GP are using examples of failed startups as basis of your criticism. One mentions "50 engineers" and I'd bet not a single one had an "engineer" bone or training in their body. So I read that as 50 blog-driven programmers fail to execute a sophisticated system design they read about on some blog. Surprise? Using Kafka as the distributed log of a deconstructed database is technically an order of magnitude more demanding than wiring MVC widgets, or using the latest paint by numbers framework, etc., because that is in fact developing a custom distributed database. Technical leadership and experience are forgotten things but they do matter. Let's not blame the stack.

simonh 6y ago
Theyre not criticising Kafka, they’re criticising misuse of it in use cases it’s not designed for.
steve_adams_86 6y ago
I said Kafka is great at what it's for - I'm not blaming it at all. I actually enjoyed working with it a lot, despite that it couldn't help my client very much. Fine tuning it to work better for them was probably one of the more fun projects I've worked on in the last year. It still wasn't the right tool for the job they were trying to do, though. They could have replaced their brokers with simple worker queues and fed the raw data and results into a db. That was all they needed. The fear was that when they got MASSIVE, that architecture wouldn't scale for them. In reality it absolutely would, but like you said, the blogs told them otherwise. Kafka is fine, using it for the wrong thing isn't.
- JamesBarney 6y ago
  If I had a dollar for every project that couldn't make a 1000 users happy because they were too worried about making a million happy.
- eternalban 6y ago
  No, you said it was a mistake to use Kafka as database. It's been pointed out that a key member of the Kafka trio, Jay Kreps, is on record that you can definitely use it as a source of truth. NYTimes example is also of a competent team that uses Kafka as a source of truth.
  - steve_adams_86 6y ago
    Fair enough, I should have been more explicit in saying that the team was using it in a weird way for their own purposes. Not only was the use case wrong for them, but the technology wasn't well understood by anyone either. The implementation was partially nonsense. I'm not criticizing Kafka so much as the way I've seen it used in software. It's largely down to user error, and like I said, Kafka is good software. I had to learn a lot about it for that project and it was a lot of fun. It was my first deep dive into message brokers and I really loved it, so I mean no disrespect to the software or its maintainers.
    - eternalban 6y ago
      For the record, I don't think any piece of software is beyond criticism, including Kafka. That said, I frankly have no idea why this sub-thread has meandered into the topic of Kafka the software! :) You could replace Kafka with Pulsar (or DistributedLog for that matter) in my comments above and it would stand. Is using a distributed log as the WAL component of a event-sourcing or distributed database entirely a bad idea? I am of the opinion that it is a viable but sophisticated approach. Peace out.

simonh 6y ago

Theyre not criticising Kafka, they’re criticising misuse of it in use cases it’s not designed for.

steve_adams_86 6y ago

I said Kafka is great at what it's for - I'm not blaming it at all. I actually enjoyed working with it a lot, despite that it couldn't help my client very much. Fine tuning it to work better for them was probably one of the more fun projects I've worked on in the last year. It still wasn't the right tool for the job they were trying to do, though. They could have replaced their brokers with simple worker queues and fed the raw data and results into a db. That was all they needed. The fear was that when they got MASSIVE, that architecture wouldn't scale for them. In reality it absolutely would, but like you said, the blogs told them otherwise. Kafka is fine, using it for the wrong thing isn't.

JamesBarney 6y ago
If I had a dollar for every project that couldn't make a 1000 users happy because they were too worried about making a million happy.
eternalban 6y ago
No, you said it was a mistake to use Kafka as database. It's been pointed out that a key member of the Kafka trio, Jay Kreps, is on record that you can definitely use it as a source of truth. NYTimes example is also of a competent team that uses Kafka as a source of truth.
- steve_adams_86 6y ago
  Fair enough, I should have been more explicit in saying that the team was using it in a weird way for their own purposes. Not only was the use case wrong for them, but the technology wasn't well understood by anyone either. The implementation was partially nonsense. I'm not criticizing Kafka so much as the way I've seen it used in software. It's largely down to user error, and like I said, Kafka is good software. I had to learn a lot about it for that project and it was a lot of fun. It was my first deep dive into message brokers and I really loved it, so I mean no disrespect to the software or its maintainers.
  - eternalban 6y ago
    For the record, I don't think any piece of software is beyond criticism, including Kafka. That said, I frankly have no idea why this sub-thread has meandered into the topic of Kafka the software! :) You could replace Kafka with Pulsar (or DistributedLog for that matter) in my comments above and it would stand. Is using a distributed log as the WAL component of a event-sourcing or distributed database entirely a bad idea? I am of the opinion that it is a viable but sophisticated approach. Peace out.