5 ms·

I think they make some fair points but I also don’t think they had a firm enough grasp on how RabbitMQ works

Fizzadar 6y ago
Came here to say this, in particular: > This problem was never root caused, though we suspect an issue in the Celery workers themselves and not RabbitMQ. So the problem wasn't Rabbit itself, but the usage/lack of understanding.
thu2111 6y ago
Could you elaborate? It's a great writeup. But I'm personally curious about them hitting the limits of vertical scalability. They said they used the biggest node 'available to them' without clarifying how big that was. If the issue was RabbitMQ hitting resource limits then upgrading its hardware (or swapping it out for some other similarly featureful broker but which is faster) might have fixed some of the problems too, albeit not things like insufficient observability. I also wonder to what extent RabbitMQ suffers from being written in Erlang. Switching to a featureful broker with sharding and replication that isn't Kafka e.g. Artemis might have allowed them to avoid rewriting code to not use helpful features. This performance blog suggests RabbitMQ may top out at ~4000 messages/sec when making things durable, which isn't especially good performance: https://softwaremill.com/mqperf/ https://softwaremill.com/mqperf/
- conradfr 6y ago
  As someone noted in the comments it's because on RabbitMQ one queue uses only one CPU core by design and the whole benchmark is based on using one queue.
- sumtechguy 6y ago
  Queue style systems usually start running into limits of the disk subsystem. Kafka is distributed log style. What that means is 'sharding' is basically built into kafka while MQ you need to build it in one layer up. Kafka can run into the same issues if you pick a poor hash for your partition. The messages per second really depends on the size of the messages too. For example in one of my tests I was hitting well over 10k-20k messages per second if the message size was 10 bytes (if my memory is working today). It dropped off dramatically as size grew. The limit was directly determined by disk and cpu. Switching to a different stack would mitigate some of that. But in the end if you expect scale (millions of messages an hour) you need a distributed system in there somewhere eventually. Would I personally reach for kafka out of the box? Not so much. I personally think it is quite a monster to get setup correctly. If you need dual datacenter up time it also becomes very expensive as you need to buy cloudera or confluent. Or have a plan to manage it yourself which will be 'interesting'.
  - rad_gruchalski 6y ago
    > If you need dual datacenter up time it also becomes very expensive as you need to buy cloudera or confluent. MirrorMaker is pretty decent and part of the oss release. There are also multiple alternatives available. There is no need to buy anything.
- nitrogen 6y ago
  I was sending over 10x that many messages per second, including serialization and deserialization, from Ruby, over a very simple RabbitMQ shovel across datacenters in two time zones, to Ruby, with disk persistence delaying acks on both ends of the shovel, in 2014.

Came here to say this, in particular: > This problem was never root caused, though we suspect an issue in the Celery workers themselves and not RabbitMQ. So the problem wasn't Rabbit itself, but the usage/lack of understanding.

thu2111 6y ago

Could you elaborate? It's a great writeup. But I'm personally curious about them hitting the limits of vertical scalability. They said they used the biggest node 'available to them' without clarifying how big that was. If the issue was RabbitMQ hitting resource limits then upgrading its hardware (or swapping it out for some other similarly featureful broker but which is faster) might have fixed some of the problems too, albeit not things like insufficient observability. I also wonder to what extent RabbitMQ suffers from being written in Erlang. Switching to a featureful broker with sharding and replication that isn't Kafka e.g. Artemis might have allowed them to avoid rewriting code to not use helpful features. This performance blog suggests RabbitMQ may top out at ~4000 messages/sec when making things durable, which isn't especially good performance: https://softwaremill.com/mqperf/ https://softwaremill.com/mqperf/

conradfr 6y ago
As someone noted in the comments it's because on RabbitMQ one queue uses only one CPU core by design and the whole benchmark is based on using one queue.
sumtechguy 6y ago
Queue style systems usually start running into limits of the disk subsystem. Kafka is distributed log style. What that means is 'sharding' is basically built into kafka while MQ you need to build it in one layer up. Kafka can run into the same issues if you pick a poor hash for your partition. The messages per second really depends on the size of the messages too. For example in one of my tests I was hitting well over 10k-20k messages per second if the message size was 10 bytes (if my memory is working today). It dropped off dramatically as size grew. The limit was directly determined by disk and cpu. Switching to a different stack would mitigate some of that. But in the end if you expect scale (millions of messages an hour) you need a distributed system in there somewhere eventually. Would I personally reach for kafka out of the box? Not so much. I personally think it is quite a monster to get setup correctly. If you need dual datacenter up time it also becomes very expensive as you need to buy cloudera or confluent. Or have a plan to manage it yourself which will be 'interesting'.
- rad_gruchalski 6y ago
  > If you need dual datacenter up time it also becomes very expensive as you need to buy cloudera or confluent. MirrorMaker is pretty decent and part of the oss release. There are also multiple alternatives available. There is no need to buy anything.
nitrogen 6y ago
I was sending over 10x that many messages per second, including serialization and deserialization, from Ruby, over a very simple RabbitMQ shovel across datacenters in two time zones, to Ruby, with disk persistence delaying acks on both ends of the shovel, in 2014.