5 ms·

RabbitMQ. It just works. We're operating with in the normal use case for it. We turn it on, transfer 50k or so messages a day, then forget about. The memory is

RabbitMQ. It just works. We're operating with in the normal use case for it. We turn it on, transfer 50k or so messages a day, then forget about. The memory is fairly low, CPU is low and constant. The connections for month or year long clients stay open.

lucb1e 10y ago
We had to use a message queueing system for school, they recommended some Oracle thing. Other systems were allowed though, and most of the people chose RabbitMQ. Can confirm, it's extremely easy to make Python, Java and PHP talk to each other. I haven't found anything it can't do yet, and it's not hogging resources either.
darkr 10y ago
I'm actually a fan of rabbitMQ generally, but I'm not sure I'd put it in the category of notoriously reliable software. It can throw away acknowledged writes (afaik https://aphyr.com/posts/315-jepsen-rabbitmq https://aphyr.com/posts/315-jepsen-rabbitmq has still not been fully fixed). 50k messages/day is pretty low throughout and under that load it should plod through just fine. Under much higher loads than that (10's - 100's millions of messages day) I've seen it die in a pretty ungraceful manner. At that scale, Kafka is a good candidate.
- busterarm 10y ago
  Under those high loads I've seen every single MQ fall over like a house of cards. At lighter loads you can get the job done with other tools that scale but with a little bit more pain. Leaves me super skeptical about them on the whole.
  - cespare 10y ago
    We push on the order of ~a billion messages / day into one of our Kafka clusters without having too much scaling difficulty.
    - jstanier 10y ago
      Indeed. We have completely replaced all queues with Kafka. In fact, I'm not sure what it would take for us to even try anything else at this point.
- AznHisoka 10y ago
  We process at least 30 million messages a day with sidekiq and had to start sharding message queues for different purposes. At some point it's not worth trying to optimize the messaging.
morgante 10y ago
In my experience, RabbitMQ is close to the opposite of reliable. When I think of reliability, I think of Postgres: it works well, and it's hard to make it not work reliably. With RabbitMQ, it's hard to make it work reliably.
donatj 10y ago
We poked RabbitMQ, Gearman, SQS, a few others, and never got the reliability we demanded. To be fair, we're handling in the order of 50 million messages a day. Ended up just writing a simple file system based queue in Golang and it's working spectacularly. Just let the SSD do the work. Couple hundred loc all told. We rarely crack 10-15% cpu and memory use sits statically at around 30mb.
- stevekemp 10y ago
  I've gone through a few things too, eventually settled on Redis. Would be interested in seeing your queue, if it were public?
  - donatj 10y ago
    I'd love to make it public, might be a while with company politics being what they are. We have a bad habit of not even sharing with other parts of the company.
MichaelGG 10y ago
50K is nothing to talk about, really. Over a workday, that's like 2 a second. Even at 5x that much, it's enough for many uncached random hard disk IOs per message.

lucb1e 10y ago

We had to use a message queueing system for school, they recommended some Oracle thing. Other systems were allowed though, and most of the people chose RabbitMQ. Can confirm, it's extremely easy to make Python, Java and PHP talk to each other. I haven't found anything it can't do yet, and it's not hogging resources either.

darkr 10y ago

I'm actually a fan of rabbitMQ generally, but I'm not sure I'd put it in the category of notoriously reliable software. It can throw away acknowledged writes (afaik https://aphyr.com/posts/315-jepsen-rabbitmq https://aphyr.com/posts/315-jepsen-rabbitmq has still not been fully fixed). 50k messages/day is pretty low throughout and under that load it should plod through just fine. Under much higher loads than that (10's - 100's millions of messages day) I've seen it die in a pretty ungraceful manner. At that scale, Kafka is a good candidate.

busterarm 10y ago
Under those high loads I've seen every single MQ fall over like a house of cards. At lighter loads you can get the job done with other tools that scale but with a little bit more pain. Leaves me super skeptical about them on the whole.
- cespare 10y ago
  We push on the order of ~a billion messages / day into one of our Kafka clusters without having too much scaling difficulty.
  - jstanier 10y ago
    Indeed. We have completely replaced all queues with Kafka. In fact, I'm not sure what it would take for us to even try anything else at this point.
AznHisoka 10y ago
We process at least 30 million messages a day with sidekiq and had to start sharding message queues for different purposes. At some point it's not worth trying to optimize the messaging.

morgante 10y ago

In my experience, RabbitMQ is close to the opposite of reliable. When I think of reliability, I think of Postgres: it works well, and it's hard to make it not work reliably. With RabbitMQ, it's hard to make it work reliably.

donatj 10y ago

We poked RabbitMQ, Gearman, SQS, a few others, and never got the reliability we demanded. To be fair, we're handling in the order of 50 million messages a day. Ended up just writing a simple file system based queue in Golang and it's working spectacularly. Just let the SSD do the work. Couple hundred loc all told. We rarely crack 10-15% cpu and memory use sits statically at around 30mb.

stevekemp 10y ago
I've gone through a few things too, eventually settled on Redis. Would be interested in seeing your queue, if it were public?
- donatj 10y ago
  I'd love to make it public, might be a while with company politics being what they are. We have a bad habit of not even sharing with other parts of the company.

MichaelGG 10y ago

50K is nothing to talk about, really. Over a workday, that's like 2 a second. Even at 5x that much, it's enough for many uncached random hard disk IOs per message.