3 ms·

I fail to understand the hype behind Kafka, my guess is that 99% of Kafka use cases can be handled by a simple RabbitMQ/Celery etc. But maybe its not considered

I fail to understand the hype behind Kafka, my guess is that 99% of Kafka use cases can be handled by a simple RabbitMQ/Celery etc. But maybe its not considered cool enough?

monstrado 9y ago
Perhaps you haven't spent enough time reading into Kafka? It solves very different problems than the RabbitMQ/Clery "worker queue" pattern you're referring to. That doesn't necessarily mean people don't use Kafka for the wrong reasons.
- aub3bhat 9y ago
  I don't deny that Kafka solves different problems and has excellent performance at high loads (100K msg/second). >> That doesn't necessarily mean people don't use Kafka for the wrong reason. Exactly what I meant.
- andy_ppp 9y ago
  This is the best explanation I've found about Kafka and Samza: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...
bsg75 9y ago
I too would be very interested in understanding more about the Kafka hype. Its everywhere lately.
- monstrado 9y ago
  Why not just go to their documentation? http://kafka.apache.org/documentation/#introduction http://kafka.apache.org/documentation/#introduction
  - bsg75 9y ago
    I can understand the product from the docs, but not necessarily why Kafka-everything seems to be approaching Hadoop-everything.
    - Marazan 9y ago
      Lots of problems can be understood as a immutable log of events. Kafka is a screamingly high performance, resilient, immutable log of events. It is kind of infectious though, once you have system implemented as Kafka-and-Samza the easiest way for them to communicate e and be communicated with is more Kafka and Samza. I am fully on board the Kafka hype train. Choo choo.
      - bsg75 9y ago
        That is useful context. Thanks!
        Marazan 9y ago
        The "Turning the database inside out" article that's linked a couple of posts up about Kafka and logs in general was the one that made me sit up and take notice and really made me 'get' Kafka and the entries stream oriented processing thing. It raises interesting questions and I've had fun producing both arguements for and against the approach.
        dualogy 9y ago
        What are your "against" fun arguments?
- theptip 9y ago
  Here's a quite compelling manifesto from Martin Kleppmann: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou... https://www.confluent.io/blog/apache-kafka-samza-and-the-unix-philosophy-of-distributed-data/ https://www.confluent.io/blog/apache-kafka-samza-and-the-uni... Some very interesting architectural ideas in there. In particular, the observation that if you turn off history pruning in Kafka, you have a consistent distributed log, which can then be piped off into whatever services need to consume it. That's appealing for cases where you want an audit trail, for example. Do most systems require that sort of thing? Absolutely not. RabbitMQ is boring tech, in the good sense; it is well understood and does its job, so IMO it's a better default option where it fits.
  - deleted 9y ago
    [deleted]
Cerium 9y ago
Agree. I looked at Kafka a while back for an IoT backend. Settled on RabbitMQ and a Ruby worker pool. It handled something like 10x our expected load on one $1k unix surplus node.
- marianov 9y ago
  Same case. Ingesting 100s or 1000s of IoT "events" per second,pipelining them and reacting. Solved with redis pub/sub and python subscribers.HAd machines responding to system interactions in no time and could prototype a new pipeline in 5 minutes. The big-data/"cognitive computing" unit of the company would have suggested using Hadoop, Kafka, IBM Watson , AI , neural networks and lots of billable man-hours to match my solution. They even looked interested in making and aquisition for a company with an event processing engine that did basically the same: pipe stuff trough processes. I recommend reading Event Processing in Action and then just building the same with your favourite pub/sub toolchain.
salex89 9y ago
As you said, usually it is enough, but people tend to (over)design and plan for scale, and Kafka does become a better solution above a limit. Shall you touch it, that's another topic. I wouldn't say that it is a rare occasion that you absolutely need Kafka, but a less frequent one, surely. Also I don't think it is a terrible miss to use it if you need a queue with reasonably high demands from it. You'll mostly waste operations efforts and development time. The end result will not differ too much. Nevertheless, the case from above is a very extreme case of over engineering and premature optimization. To the limit where would qualify it as reckless.
- andy_ppp 9y ago
  Why does everyone seem to think Kafka is a queue replacement; it's a high performance immutable log with a guaranteed CAP profile and the possibility to add materised views to the log. See here for details: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...
sidlls 9y ago
A good number of the cases Kafka is used for where I work are even simpler: they'd be best handled by a very tiny flask (or similar) server backed by a common data store.
buro9 9y ago
You are probably right. But then you come across the 1% case and Kafka is the only thing that you can throw at the problem that works. Not sure of exact numbers, but I think we're doing 4-5m writes per second across one Kafka cluster, and around 1m writes per second against another. Most people don't have big data, but some people do.
- cookiecaper 9y ago
  Another caveat on this. Some people are forced into "big data" solutions not because their data is inherently "big data", but because their applications output data with wanton disregard for its arrangement or structure, making it so only systems designed to manage massive streams can readily handle it. Many databases that I regularly touch would shrink an order of magnitude or more if someone went through, redid the layout, and scripted a process to perform the migration.
- e12e 9y ago
  > Not sure of exact numbers, but I think we're doing 4-5m writes per second across one Kafka cluster, and around 1m writes per second against another. I'm not saying that's the case with your system, but my immediate thought when I see those numbers is: how much of the value from those 4m/s writes could you get with a system that did something like 100 w/s? Either through sampling/statistical methods, reduce "write amplification" or simply looking hard at what is written and how often :-)
lmm 9y ago
What makes RabbitMQ/Celery/etc. any "simple"r than Kafka? I found the experience very similar, and in a JVM-oriented environment Kafka was easier to deploy.
Xylakant 9y ago
I've come to know Kafka as very reliable and well understandable queue that solves a few things much nicer than rabbitmq - for example the ability to replay all events still retained in the cluster. My primary pain with it is running the zk cluster, but other than that it's my to-go queue for moderate or larger throughput queues where loosely ordered at least once delivery is fine. My reason to pick it is the feature set and that it has yet to let me down and munch my data.
deleted 9y ago
[deleted]

I fail to understand the hype behind Kafka, my guess is that 99% of Kafka use cases can be handled by a simple RabbitMQ/Celery etc. But maybe its not considered cool enough?

monstrado 9y ago

Perhaps you haven't spent enough time reading into Kafka? It solves very different problems than the RabbitMQ/Clery "worker queue" pattern you're referring to. That doesn't necessarily mean people don't use Kafka for the wrong reasons.

aub3bhat 9y ago
I don't deny that Kafka solves different problems and has excellent performance at high loads (100K msg/second). >> That doesn't necessarily mean people don't use Kafka for the wrong reason. Exactly what I meant.
andy_ppp 9y ago
This is the best explanation I've found about Kafka and Samza: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...

bsg75 9y ago

I too would be very interested in understanding more about the Kafka hype. Its everywhere lately.

monstrado 9y ago
Why not just go to their documentation? http://kafka.apache.org/documentation/#introduction http://kafka.apache.org/documentation/#introduction
- bsg75 9y ago
  I can understand the product from the docs, but not necessarily why Kafka-everything seems to be approaching Hadoop-everything.
  - Marazan 9y ago
    Lots of problems can be understood as a immutable log of events. Kafka is a screamingly high performance, resilient, immutable log of events. It is kind of infectious though, once you have system implemented as Kafka-and-Samza the easiest way for them to communicate e and be communicated with is more Kafka and Samza. I am fully on board the Kafka hype train. Choo choo.
    - bsg75 9y ago
      That is useful context. Thanks!
      - Marazan 9y ago
        The "Turning the database inside out" article that's linked a couple of posts up about Kafka and logs in general was the one that made me sit up and take notice and really made me 'get' Kafka and the entries stream oriented processing thing. It raises interesting questions and I've had fun producing both arguements for and against the approach.
        dualogy 9y ago
        What are your "against" fun arguments?
theptip 9y ago
Here's a quite compelling manifesto from Martin Kleppmann: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou... https://www.confluent.io/blog/apache-kafka-samza-and-the-unix-philosophy-of-distributed-data/ https://www.confluent.io/blog/apache-kafka-samza-and-the-uni... Some very interesting architectural ideas in there. In particular, the observation that if you turn off history pruning in Kafka, you have a consistent distributed log, which can then be piped off into whatever services need to consume it. That's appealing for cases where you want an audit trail, for example. Do most systems require that sort of thing? Absolutely not. RabbitMQ is boring tech, in the good sense; it is well understood and does its job, so IMO it's a better default option where it fits.
- deleted 9y ago
  [deleted]

Cerium 9y ago

Agree. I looked at Kafka a while back for an IoT backend. Settled on RabbitMQ and a Ruby worker pool. It handled something like 10x our expected load on one $1k unix surplus node.

marianov 9y ago
Same case. Ingesting 100s or 1000s of IoT "events" per second,pipelining them and reacting. Solved with redis pub/sub and python subscribers.HAd machines responding to system interactions in no time and could prototype a new pipeline in 5 minutes. The big-data/"cognitive computing" unit of the company would have suggested using Hadoop, Kafka, IBM Watson , AI , neural networks and lots of billable man-hours to match my solution. They even looked interested in making and aquisition for a company with an event processing engine that did basically the same: pipe stuff trough processes. I recommend reading Event Processing in Action and then just building the same with your favourite pub/sub toolchain.

salex89 9y ago

As you said, usually it is enough, but people tend to (over)design and plan for scale, and Kafka does become a better solution above a limit. Shall you touch it, that's another topic. I wouldn't say that it is a rare occasion that you absolutely need Kafka, but a less frequent one, surely. Also I don't think it is a terrible miss to use it if you need a queue with reasonably high demands from it. You'll mostly waste operations efforts and development time. The end result will not differ too much. Nevertheless, the case from above is a very extreme case of over engineering and premature optimization. To the limit where would qualify it as reckless.

andy_ppp 9y ago
Why does everyone seem to think Kafka is a queue replacement; it's a high performance immutable log with a guaranteed CAP profile and the possibility to add materised views to the log. See here for details: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...

sidlls 9y ago

A good number of the cases Kafka is used for where I work are even simpler: they'd be best handled by a very tiny flask (or similar) server backed by a common data store.

buro9 9y ago

You are probably right. But then you come across the 1% case and Kafka is the only thing that you can throw at the problem that works. Not sure of exact numbers, but I think we're doing 4-5m writes per second across one Kafka cluster, and around 1m writes per second against another. Most people don't have big data, but some people do.

cookiecaper 9y ago
Another caveat on this. Some people are forced into "big data" solutions not because their data is inherently "big data", but because their applications output data with wanton disregard for its arrangement or structure, making it so only systems designed to manage massive streams can readily handle it. Many databases that I regularly touch would shrink an order of magnitude or more if someone went through, redid the layout, and scripted a process to perform the migration.
e12e 9y ago
> Not sure of exact numbers, but I think we're doing 4-5m writes per second across one Kafka cluster, and around 1m writes per second against another. I'm not saying that's the case with your system, but my immediate thought when I see those numbers is: how much of the value from those 4m/s writes could you get with a system that did something like 100 w/s? Either through sampling/statistical methods, reduce "write amplification" or simply looking hard at what is written and how often :-)

lmm 9y ago

What makes RabbitMQ/Celery/etc. any "simple"r than Kafka? I found the experience very similar, and in a JVM-oriented environment Kafka was easier to deploy.

Xylakant 9y ago

I've come to know Kafka as very reliable and well understandable queue that solves a few things much nicer than rabbitmq - for example the ability to replay all events still retained in the cluster. My primary pain with it is running the zk cluster, but other than that it's my to-go queue for moderate or larger throughput queues where loosely ordered at least once delivery is fine. My reason to pick it is the feature set and that it has yet to let me down and munch my data.

deleted 9y ago

[deleted]