4 ms·

The thing I've found about message queues is that it's pretty unimportant what platform they're based on. I used to be tempted to go with ActiveMQ "because it

The thing I've found about message queues is that it's pretty unimportant what platform they're based on. I used to be tempted to go with ActiveMQ "because it runs on the JVM", but that turns out to be pretty irrelevant. No matter what your queue provider, you're most likely just using TCP sockets anyway, so don't think you need to be using Erlang just to take advantage of RabbitMQ. Yes, this means the MQ can basically be thought of as a black box. But that's also kind of the point.

marshray 15y ago
What I've seen with RabbitMQ is that the memory usage can grow and grow. If it runs out of either committed RAM or address space, it crashes. I suspect this may be a feature of the open source Erlang VM that's improved in the pay-for-support version. Even though RabbitMQ pays the performance cost of bouncing everything off the disk, it's not quite recoverable from that type of crash. I'm sure my information is old, perhaps these things are improved in newer versions. Still, the behavior of the Erlang VM itself was relevant. As a C++ programmer, I wouldn't feel comfortable abstracting away memory management for this type of server which is both so memory intensive and requires high reliability. But of course many people swear by highly reliable JVM based stuff, so perhaps I'm wrong about it. :-)
- deleted 15y ago
  [deleted]
- FooBarWidget 15y ago
  I've used RabbitMQ 2.4 in production with queue lengths of several hundred thousand. Each queue item can be as large as 100 KB. RabbitMQ's memory usage never went beyond ~150 MB.
- rbranson 15y ago
  Actually Erlang/OTP provides exactly the type of abstraction layer that's extremely useful for RabbitMQ's use case. It's specifically designed for consistent latency, predictable memory usage, and resilience under failure conditions. You know, basically the things that make an MQ worth using. It was true that previously RabbitMQ could run out of memory and be unable to take more messages, but that changed in 2.0 and it will now swap messages to disk. Of course, then you pay the consequences of slow disks and can always run out of disk space, but nothing is invincible.
- gcampbell 15y ago
  RabbitMQ has gotten much better about memory management as of version 2.0.
- tsuraan 15y ago
  What I've seen with RabbitMQ is that the memory usage can grow and grow Prior to 2.0, the contents of Rabbit's queues were always in RAM. Even durable queues with persistent messages always had their contents mirrored in RAM. As of 2.0, Rabbit can store queue contents on disk or in RAM, so as it faces memory pressure, it will send data to disk. If the disk is too slow for it to relieve its pressure, it will block connections that have sent data in the past until it can free up enough memory to continue. Last I checked, Rabbit does by default keep the associative map between queues and message in RAM, so its memory usage isn't technically bounded (it's a around a dozen bytes per message, IIRC). If that's a problem, the toke plugin uses tokyo cabinet to store the associations, which I believe allows Rabbit to have entirely bounded memory use. I also haven't checked on that for a while, so it may be that toke is installed by default now.
- rabbitmq 15y ago
  "a feature of the open source Erlang VM that's improved in the pay-for-support version" Erlang was created by a telco in the 1990s and since then has been battered in production by serious users. It's real software. The version that Ericsson provide as open source is not crippleware as you seem to imply. "RabbitMQ pays the performance cost of bouncing everything off the disk" No it doesn't. RabbitMQ only uses the disk if tell it to do so, eg if you require messages to be persisted when they cannot be delivered immediately. "memory usage can grow and grow" If you stuff data into a messaging server without draining it on the consumer side, then memory usage will grow. This will also happen if you write your messaging system in C++. There are two solutions to this problem: - flow control, where you tell producers to back off - paging to disk, where you flush data from memory when it is on the disk Neither of these is trivial to implement which is why there is a big gap between toy messaging systems and serious products. As others point out on this page, RabbitMQ has support for both these features. In particular a lot of memory management capability has been added since 2.0. I hope this helps.

marshray 15y ago

What I've seen with RabbitMQ is that the memory usage can grow and grow. If it runs out of either committed RAM or address space, it crashes. I suspect this may be a feature of the open source Erlang VM that's improved in the pay-for-support version. Even though RabbitMQ pays the performance cost of bouncing everything off the disk, it's not quite recoverable from that type of crash. I'm sure my information is old, perhaps these things are improved in newer versions. Still, the behavior of the Erlang VM itself was relevant. As a C++ programmer, I wouldn't feel comfortable abstracting away memory management for this type of server which is both so memory intensive and requires high reliability. But of course many people swear by highly reliable JVM based stuff, so perhaps I'm wrong about it. :-)

deleted 15y ago
[deleted]
FooBarWidget 15y ago
I've used RabbitMQ 2.4 in production with queue lengths of several hundred thousand. Each queue item can be as large as 100 KB. RabbitMQ's memory usage never went beyond ~150 MB.
rbranson 15y ago
Actually Erlang/OTP provides exactly the type of abstraction layer that's extremely useful for RabbitMQ's use case. It's specifically designed for consistent latency, predictable memory usage, and resilience under failure conditions. You know, basically the things that make an MQ worth using. It was true that previously RabbitMQ could run out of memory and be unable to take more messages, but that changed in 2.0 and it will now swap messages to disk. Of course, then you pay the consequences of slow disks and can always run out of disk space, but nothing is invincible.
gcampbell 15y ago
RabbitMQ has gotten much better about memory management as of version 2.0.
tsuraan 15y ago
What I've seen with RabbitMQ is that the memory usage can grow and grow Prior to 2.0, the contents of Rabbit's queues were always in RAM. Even durable queues with persistent messages always had their contents mirrored in RAM. As of 2.0, Rabbit can store queue contents on disk or in RAM, so as it faces memory pressure, it will send data to disk. If the disk is too slow for it to relieve its pressure, it will block connections that have sent data in the past until it can free up enough memory to continue. Last I checked, Rabbit does by default keep the associative map between queues and message in RAM, so its memory usage isn't technically bounded (it's a around a dozen bytes per message, IIRC). If that's a problem, the toke plugin uses tokyo cabinet to store the associations, which I believe allows Rabbit to have entirely bounded memory use. I also haven't checked on that for a while, so it may be that toke is installed by default now.
rabbitmq 15y ago
"a feature of the open source Erlang VM that's improved in the pay-for-support version" Erlang was created by a telco in the 1990s and since then has been battered in production by serious users. It's real software. The version that Ericsson provide as open source is not crippleware as you seem to imply. "RabbitMQ pays the performance cost of bouncing everything off the disk" No it doesn't. RabbitMQ only uses the disk if tell it to do so, eg if you require messages to be persisted when they cannot be delivered immediately. "memory usage can grow and grow" If you stuff data into a messaging server without draining it on the consumer side, then memory usage will grow. This will also happen if you write your messaging system in C++. There are two solutions to this problem: - flow control, where you tell producers to back off - paging to disk, where you flush data from memory when it is on the disk Neither of these is trivial to implement which is why there is a big gap between toy messaging systems and serious products. As others point out on this page, RabbitMQ has support for both these features. In particular a lot of memory management capability has been added since 2.0. I hope this helps.