4 ms·

Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast m

Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.

rafaelturk 3y ago
If your have a clean event-driven architecture, ie messages are completely agnostic and decoupled from one-another you don't need Kafka.
- ceencee 3y ago
  This is a ridiculous statement if you really build an EDA. Kafka is what enables the decoupling.
- inkyoto 3y ago
  Event-driven architecture is an architectural principle, and Kafka, RabbitMQ/ActiveMQ, Pulsar, NATS and so forth are implementations that support the event-driven architectural principle. Yet, all of them range in a variety, complexity and extent of features they provide which may or may not be a good fit for a particular use case. Traditional message brokers (RabbitMQ and similar) do support the event-driven architecture, yet the data they handle is ephemeral. Once a message has been processed, it is gone forever. Connecting a new raw data source is not an established practice and requires a technical «adapter» of sorts to be built. High concurrency levels is problematic for scenarios where the strict message processing ordering is required: the traditional message brokers do not handle it well in highly parallel scenarios out of the box. Kafka and similar also support event-driven architectures, yet they allow the data to be processed multiple times – by existing (i.e. a data replay) and, most importantly, new or unknown at the time consumers (note: this is distinct from the data replay!). This is allows to plug existing data source(s) into a data streaming platform (Kafka) and incrementally add new data consumers and processors over the time with the datasets being available intact. This is an important distinction. Kafka and similar also improve on the strict processing order guarantee by allowing a message source (a Kafka topic) to be explicitly partitioned out and guaranteeing that the message order will be retained and enforced for a consumer group receiving messages from that partition. To recap, traditional message brokers are a good fit for handling the ephemeral data, and data streaming platforms are a good fit for connecting data sources and allowing the data to be ingested multiple times. Both implement and support event-driven architectures in a variety of scenarios.
  - rcombatwombat 3y ago
    NATS with JetStream provides _both_ queuing like a traditional message broker and multiple data replay from offset (plus KV, and request/reply)
ceencee 3y ago
Pretty much every bank uses kafka ad the central messaging layer. What people are missing in almost every post here is the write once read many without data duplication and with different offsets is the killer app for Kafka beyond just the near infinite scale which is also super appealing. The failure modes are way way better than Rabbit as well. Note: I owned the streaming platform for a top 5 bank in the us.
- leetbulb 3y ago
  Yeah, I'm sorry to others, but if you require the guarantees and compliance that Kafka provides, Kafka wins, especially at this kind of scale. I'd love to see RabbitMQ scaled out to handle hundreds of trillions of events per day and able to retain years worth of highly durable, immutable, and replayable event storage. Ultimately, this comparison is apples vs oranges...
dalyons 3y ago
Rabbitmq sucks to scale. clustering and partitioning were terrible for a long time, maybe still is. Clusters dying in split brained ways, nodes crashing terribly / unrecoverable if they exceed iops or storage limits. You couldn’t pay me enough to run a high volume rMQ cluster again. Never mind that the persistent durable log pattern of kakfa enables a lot replay type use cases that are very beneficial in financial systems specifically. It’s not solely because of hype at all, it’s objectively better for many use cases.

rafaelturk 3y ago

If your have a clean event-driven architecture, ie messages are completely agnostic and decoupled from one-another you don't need Kafka.

ceencee 3y ago
This is a ridiculous statement if you really build an EDA. Kafka is what enables the decoupling.
inkyoto 3y ago
Event-driven architecture is an architectural principle, and Kafka, RabbitMQ/ActiveMQ, Pulsar, NATS and so forth are implementations that support the event-driven architectural principle. Yet, all of them range in a variety, complexity and extent of features they provide which may or may not be a good fit for a particular use case. Traditional message brokers (RabbitMQ and similar) do support the event-driven architecture, yet the data they handle is ephemeral. Once a message has been processed, it is gone forever. Connecting a new raw data source is not an established practice and requires a technical «adapter» of sorts to be built. High concurrency levels is problematic for scenarios where the strict message processing ordering is required: the traditional message brokers do not handle it well in highly parallel scenarios out of the box. Kafka and similar also support event-driven architectures, yet they allow the data to be processed multiple times – by existing (i.e. a data replay) and, most importantly, new or unknown at the time consumers (note: this is distinct from the data replay!). This is allows to plug existing data source(s) into a data streaming platform (Kafka) and incrementally add new data consumers and processors over the time with the datasets being available intact. This is an important distinction. Kafka and similar also improve on the strict processing order guarantee by allowing a message source (a Kafka topic) to be explicitly partitioned out and guaranteeing that the message order will be retained and enforced for a consumer group receiving messages from that partition. To recap, traditional message brokers are a good fit for handling the ephemeral data, and data streaming platforms are a good fit for connecting data sources and allowing the data to be ingested multiple times. Both implement and support event-driven architectures in a variety of scenarios.
- rcombatwombat 3y ago
  NATS with JetStream provides _both_ queuing like a traditional message broker and multiple data replay from offset (plus KV, and request/reply)

ceencee 3y ago

Pretty much every bank uses kafka ad the central messaging layer. What people are missing in almost every post here is the write once read many without data duplication and with different offsets is the killer app for Kafka beyond just the near infinite scale which is also super appealing. The failure modes are way way better than Rabbit as well. Note: I owned the streaming platform for a top 5 bank in the us.

leetbulb 3y ago
Yeah, I'm sorry to others, but if you require the guarantees and compliance that Kafka provides, Kafka wins, especially at this kind of scale. I'd love to see RabbitMQ scaled out to handle hundreds of trillions of events per day and able to retain years worth of highly durable, immutable, and replayable event storage. Ultimately, this comparison is apples vs oranges...

dalyons 3y ago

Rabbitmq sucks to scale. clustering and partitioning were terrible for a long time, maybe still is. Clusters dying in split brained ways, nodes crashing terribly / unrecoverable if they exceed iops or storage limits. You couldn’t pay me enough to run a high volume rMQ cluster again. Never mind that the persistent durable log pattern of kakfa enables a lot replay type use cases that are very beneficial in financial systems specifically. It’s not solely because of hype at all, it’s objectively better for many use cases.