6 ms·

Must be something about Kafka to attract these kind of explanations. Another one few months back was a children's book on Kafka [1] . For me it just look like s

by geodel 5y ago

Must be something about Kafka to attract these kind of explanations. Another one few months back was a children's book on Kafka [1] . For me it just look like solution looking for actual problems. I wonder if Kafka represents an existential angst in these Kubernetized Microservice times. Or is it more simply I am just too dumb to learn and use this shit correctly. 1. https://news.ycombinator.com/item?id=27541339 https://news.ycombinator.com/item?id=27541339

Hamuko 5y ago
The otters definitely overengineered the living crap out of their party notifications.
- geodel 5y ago
  Turns out some otters did have doubt. But "if Linkedin use Kafka, it must be Good" clinched the deal.
joking 5y ago
the same can be said about microservices.
deleted 5y ago
[deleted]
bimboler38 5y ago
This is exactly what I thought. Maybe it's because it's popular or maybe it's because the docs are notoriously bad. Or both.
berkes 5y ago
When you are wondering whether you might need Kafka, it is certain that you don't need it. But there are times when you have a problem, and amongst the possible solutions is Kafka. I've come across Kafkaesque problems only three times in the last seven years: a hosting platform that had to parse logs of over 700 WordPress sites for security and other businesslogic. Putting all events of a financial app backend into datalakes and filtering and parsing all openstreetmap changesets live.
- benjaminwootton 5y ago
  Not sure I agree. It seems as good a way as any to decouple systems, asynchronously exchange messages between services, get them into durable storage, exactly once processing, replay semantics etc. I think it should be on the table at least whenever we have two services needing to exchange data. Maybe a few use cases could be switched out for direct API calls, but I think Kafka hits the sweet spot in many situations. What alternatives would you be looking at?
  - weego 5y ago
    It is as good as any if you have no financial constraints and no technical overhead and time constraints, but everyone does. Kafka is one of those systems that needs to be justified by out-scaling other solutions that don't come wedded with all its baggage.
    - hansbo 5y ago
      What would you say is its baggage?
      - berkes 5y ago
        For me, the baggage is mostly the complexity of the service. With that comes monitoring, maintenance, tuning, debugging and troubleshooting. Lessened somewhat with SaaS products like Amazon Kinesis (technically not a Kafka, but close). Another "baggage" is that an event-driven setup is eventual-consistent -and async- by nature. If your software already is eventual-consistent, this is not a problem. But it is a huge change if you come from a blocking/simple "crud" setup.
        piyh 5y ago
        I second Kafka having massive operational overhead. It's a burden and is killing any support for it within our org.
  - berkes 5y ago
lmm 5y ago
Sometimes I wonder if I'm the crazy one. Kafka seems to me to be the only sensible foundational datastore out there: it can maintain and propagate a log with all the properties you would want a datastore to have. Relational database seem to be a crazily overengineered solution in search of a problem, with incredibly poor reliability properties (essentially none of them are true master-master HA out of the box, and they tend to require significant compromises to make them so) to boot.
- valenterry 5y ago
  Both are good for different purposes. Want to store data, query it in arbitrary and hard to foresee ways and also want to easily tune performance for these queries? Relational datastore it is. Want to have ACID? Well, relational datastore it is. Kafka is not the right solution for these problems.
- dmitriid 5y ago
  > Relational database seem to be a crazily overengineered solution in search of a problem I mean, whoever in their right mind would want to: - have a snapshot of data - query data, including ad-hoc querying - query related data - have trasactional updates to data When all you need is an unbounded stream of data that you need to traverse in order to do all these things.
  - lmm 5y ago
    > - have a snapshot of data Being able to see a snapshot is good, and I would hope to see a higher-level abstraction that can offer that on top of something Kafka-like. But making the current state the primary thing is a huge step backwards, especially when you don't get a history at all by default. > - query data, including ad-hoc querying OK, fair, ad-hoc queries are one thing that relational databases are legitimately good at. Something that can maintain secondary indicies and do query planning based on them is definitely useful. But you're asking for trouble if you use them in your live dataflow or allow ad-hoc queries to write to your datastore. > - have trasactional updates to data I do think this one is genuinely a mistake. What do you do when a transaction fails? All of the answers I've heard imply that you didn't actually need transactions in the first place.
sivakon 5y ago
In our org, all the website events are pushed into Kafka and it enabled all the teams to develop real-time event driven applications using these events without needing any access to website database or any infra that handles the main website. It’s truly a blessing in large organisations.

by geodel 5y ago

Hamuko 5y ago

The otters definitely overengineered the living crap out of their party notifications.

geodel 5y ago
Turns out some otters did have doubt. But "if Linkedin use Kafka, it must be Good" clinched the deal.

joking 5y ago

the same can be said about microservices.

deleted 5y ago

[deleted]

bimboler38 5y ago

This is exactly what I thought. Maybe it's because it's popular or maybe it's because the docs are notoriously bad. Or both.

berkes 5y ago

When you are wondering whether you might need Kafka, it is certain that you don't need it. But there are times when you have a problem, and amongst the possible solutions is Kafka. I've come across Kafkaesque problems only three times in the last seven years: a hosting platform that had to parse logs of over 700 WordPress sites for security and other businesslogic. Putting all events of a financial app backend into datalakes and filtering and parsing all openstreetmap changesets live.

benjaminwootton 5y ago
Not sure I agree. It seems as good a way as any to decouple systems, asynchronously exchange messages between services, get them into durable storage, exactly once processing, replay semantics etc. I think it should be on the table at least whenever we have two services needing to exchange data. Maybe a few use cases could be switched out for direct API calls, but I think Kafka hits the sweet spot in many situations. What alternatives would you be looking at?
- weego 5y ago
  It is as good as any if you have no financial constraints and no technical overhead and time constraints, but everyone does. Kafka is one of those systems that needs to be justified by out-scaling other solutions that don't come wedded with all its baggage.
  - hansbo 5y ago
    What would you say is its baggage?
    - berkes 5y ago
      For me, the baggage is mostly the complexity of the service. With that comes monitoring, maintenance, tuning, debugging and troubleshooting. Lessened somewhat with SaaS products like Amazon Kinesis (technically not a Kafka, but close). Another "baggage" is that an event-driven setup is eventual-consistent -and async- by nature. If your software already is eventual-consistent, this is not a problem. But it is a huge change if you come from a blocking/simple "crud" setup.
      - piyh 5y ago
        I second Kafka having massive operational overhead. It's a burden and is killing any support for it within our org.
- berkes 5y ago

lmm 5y ago

Sometimes I wonder if I'm the crazy one. Kafka seems to me to be the only sensible foundational datastore out there: it can maintain and propagate a log with all the properties you would want a datastore to have. Relational database seem to be a crazily overengineered solution in search of a problem, with incredibly poor reliability properties (essentially none of them are true master-master HA out of the box, and they tend to require significant compromises to make them so) to boot.

valenterry 5y ago
Both are good for different purposes. Want to store data, query it in arbitrary and hard to foresee ways and also want to easily tune performance for these queries? Relational datastore it is. Want to have ACID? Well, relational datastore it is. Kafka is not the right solution for these problems.
dmitriid 5y ago
> Relational database seem to be a crazily overengineered solution in search of a problem I mean, whoever in their right mind would want to: - have a snapshot of data - query data, including ad-hoc querying - query related data - have trasactional updates to data When all you need is an unbounded stream of data that you need to traverse in order to do all these things.
- lmm 5y ago
  > - have a snapshot of data Being able to see a snapshot is good, and I would hope to see a higher-level abstraction that can offer that on top of something Kafka-like. But making the current state the primary thing is a huge step backwards, especially when you don't get a history at all by default. > - query data, including ad-hoc querying OK, fair, ad-hoc queries are one thing that relational databases are legitimately good at. Something that can maintain secondary indicies and do query planning based on them is definitely useful. But you're asking for trouble if you use them in your live dataflow or allow ad-hoc queries to write to your datastore. > - have trasactional updates to data I do think this one is genuinely a mistake. What do you do when a transaction fails? All of the answers I've heard imply that you didn't actually need transactions in the first place.

sivakon 5y ago

In our org, all the website events are pushed into Kafka and it enabled all the teams to develop real-time event driven applications using these events without needing any access to website database or any infra that handles the main website. It’s truly a blessing in large organisations.