6 ms·

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In m

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika library was replaced by 3 lines of python and redis server. One day rabbitmq died and it tooks sys admins few weeks to get it back running. In that time I deployed my simple redis list and never looked back. To this day redis solution works without any friction whatsoever with fraction of resources. The rabbits AMQP exchange model is severely flawed and convoluted. It's the worst example of corporate software where everything works and doesn't work at the same time. I wouldn't recommend rabbitmq to my worst enemy yet there's still something attractive about it. Maybe there's a sane alternative? Maybe zeromq?

formerlurker2 6y ago
I'd be interested to know what the 3 lines of python were and also more details about the redis server you deployed to replace RabbitMQ during the outage.
- kabacha 6y ago
  I'm using a bit of a hyperbole but using redis list as a task queue is a simple as: while True: msg = r.lpop(key) try: result = do_something(msg) except Exception as e: log.error(f'failure for msg "{msg}" got {e} back to queue {key}') r.rpush(key, msg) continue Pop a member from a list, if failed plop it back to the end. It's simple, explicit and just works™
  - lbruder 6y ago
    It just works, until PUSHing the member back fails. At least use RPOPLPUSH and a separate worker queue to make sure you don't accidentally drop packets. Don't get me wrong, I'm using Redis queues myself and trying to get rid of the last remnants of RabbitMQ in our code, but there are use cases where RabbitMQ means less thinking about stuff, because it just works(tm)...
    - jlouis 6y ago
      The other classic variant here is when msg is poisonous. Then pushing it back onto the queue means it'll go again at some point, poisoning your system again. In practice, you need some dead-lettering with a message timeout. Something RabbitMQ does provide out of the box, for all its complexity.
  - ayush--s 6y ago
    couple of problems with this: (1) RPUSH can fail (2) LPOP's result might never reach your client after redis has executed it.
Mandatum 6y ago
brew install rabbitmq? I agree rabbitmq is often adopted in places it shouldn't, or ill-configured.. But to completely get rid of it because you didn't take the time to RTFM when setting it up seems a little extreme. And redis/rabbitmq have completely different use-cases 80% of the time. Sounds like you were trying to get drunk on kombucha.
- dragonsh 6y ago
  You are right in our case redis failed spectacularly with celery queue when tasks in queue exceeded the memory of the machine. It took weeks to diagnose as error message were inconsistent and setting up a Message Queue cluster with redis is a nightmare. Went back again to tried and tested rabbitMQ and it works so well. Also adding new nodes and removing nodes so easy, just used ansible to setup Erlang cookie and connect the node (thanks to OTP and BEAM). The best part is for important task queue for which we cannot accept failure we built a mechanism for fault tolerance. When you work with high availability and fault tolerant queue rabbitMQ is so good, it can recover from hardware or even VM failures, can’t say the same for redis which was a nightmare even with redis cluster.
oauea 6y ago
Odd, I've been running it for years and it's been flawless. Integrating new clients is a breeze using their, or any amqp, library.
Lazare 6y ago
If you were capable of replacing your use of rabbitmq with 3 lines of python and a redis instance, you shouldn't have been using rabbitmq to start with. That doesn't mean redis is a drop in replacement for any of the valid uses for rabbitmq though...
- lowwave 6y ago
  can you share some of those feature where rabbitmq that is useful? Tks.
mister_hn 6y ago
With docker it takes 5 seconds to set a rabbitmq instance locally.
codeduck 6y ago
Amusingly, RabbitMQ is the one message broker that HASN'T given me grief in my career.
elisharobinson 6y ago
Your fault is using pika. I thought the same untill I tried some other python libraries . Try Oslo messaging . It's used in openstack (enough said).
st1ck 6y ago
Does your solution handle the situation when consumer crashes and queue has to be accumulated (while RAM allows) until consumer is up again (maybe 1 day later)? AFAIK ZMQ can't guarantee this.
- twic 6y ago
  It must be emphasised that, despite the name, ZeroMQ is not a message queue. It is a networking library. The current blurb says: > ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. And the old meme was "ZeroMQ is a replacement for Berkeley sockets". It's a pretty cool networking library. But it makes no sense to think of it in the same slot as RabbitMQ, or even Redis. That the GP mentioned it does make me wonder if they don't really understand what they're doing. EDIT I love that the official guide actually has this diagram in! Pieter Hintjens's death was a sad loss. http://zguide.zeromq.org/page:all#How-It-Began http://zguide.zeromq.org/page:all#How-It-Began
  - jen20 6y ago
    As an aside, the ZeroMQ Guide is the finest piece of technical writing I have ever seen - closely followed by Pieter’s (sadly unfinished) “Scalable C” book. His non-technical writing is also excellent.
    - francislavoie 6y ago
      I had a fun time implementing the Paranoid Pirate pattern with my coworker a few years back. He wrote the server part in Python, I wrote the client in PHP. We essentially built it as a wrapper to run some C code our boss wrote that we didn't want to write a PHP extension for - we used Python as a broker to allow for some concurrency. Worked super well.
- ayush--s 6y ago
  a Redis list (what he's using) would handle this easily (I think Redis' PubSub also handles this)
ayush--s 6y ago
rabbitmq used to be one of the hardest things to self-host (even at relatively small scale). But it has been pretty stable since moving to Cloudamqp as a managed solution.
- MaxBarraclough 6y ago
  I had the same experience. I had particular trouble configuring encryption for connections. It didn't help that they'd switched configuration file-formats, but the documentation and tutorials still used the old format. Never had any stability issues though. Once it was up and running, it was solid.
smoe 6y ago
Sounds to me like the problem is, that the company taking out the cannon to shoot at sparrows and that nobody seem to have bothered to provide working dev environments? It feels common that people think they their every problem requires a planetary scale solution or one that handles every conceivable case that could occur before the heat death of the universe. Be that because your a startup and think you are going to need to support a hundred million concurrent users on launch day, or because your an enterprise and think because your are oh so important you need to use the same tools other important companies use. My own anecdote: When we refactored our early stage app we had a huge mess with the Redis based queue system. It was one of the biggest sources of errors and a massive pain to troubleshoot or even to monitor what is going on. So we investigated in a bunch of different solutions including all the usual contenders, and we ended up with: Let's just ditch the messages / queues altogether for now and just do boring old cron like jobs invoking internal api endpoints in regular intervals. This made 9 out of 10 cases a lot easier to maintain, and for the remainder, while temporary more difficult, we introduced a queue again at a much later stage. I'm not saying you shouldn't use and you can't benefit of RabbitMQ at small scales or any technology for that matter. But I think too often in tech decision making ones own or the companies perceived importance, what is cool or what would look good on a CV takes precedence over what really fits the problem in context.
kazinator 6y ago
ZeroMQ compiles to well over 600 kilobytes of machine code (just on 32 bit x86). $ size /usr/lib/i386-linux-gnu/libzmq.so text data bss dec hex filename 662134 12952 24 675110 a4d26 /usr/lib/i386-linux-gnu/libzmq.so Yikes, is there an operating system kernel with virtual memory, USB support, a few filesystems and a TCP/IP stack hiding in there? It's like 35% of glibc: $ size /lib/i386-linux-gnu/libc.so.6 text data bss dec hex filename 1917549 11624 11112 1940285 1d9b3d /lib/i386-linux-gnu/libc.so.6 Which is itself heavily bloated, trying to provide as much POSIX as it can.
mcsoft 6y ago
We used both zeromq and its sort-of-descendant nanomsg but eventually moved to NATS because of it's approach to high availability (every node knows the topology of the whole cluster) and zero management overhead. Some criticize NATS for the absence of message durability in the core technology, but we figured out we can drop this requirement in 95% of cases. Your microservices should be highly available anyway, so there's always a live consumer - and it's better to handle the message to it directly rather than introducing the overhead of storing a message somewhere and dealing with more-than-once delivery. There's a bit more to that like you should let your microservices exit gracefully and finish processing consumed messages. And of course in 5% of cases, where you can't allow to lose a single message, you have to use NATS Streaming or the likes, but so far we've been greatly impressed by NATS for high load.
nathan-io 6y ago
I'm equally surprised by your anecdote. I found it quite simple to install RabbitMQ server and its admin panel in my WSL local dev environment. And the cloud/prod instance took a few clicks (just spun up a DO Marketplace server image) followed by < five minutes of RabbitMQ user and firewall configuration. It was also dead simple to start using RabbitMQ within my application. I found a well maintained package, installed it, edited a couple lines of my application's config, and everything just worked. I specifically avoided Redis based on my understanding that it can't guarantee message persistence, so if it crashes, your unprocessed messages are lost.

formerlurker2 6y ago

I'd be interested to know what the 3 lines of python were and also more details about the redis server you deployed to replace RabbitMQ during the outage.

kabacha 6y ago
I'm using a bit of a hyperbole but using redis list as a task queue is a simple as: while True: msg = r.lpop(key) try: result = do_something(msg) except Exception as e: log.error(f'failure for msg "{msg}" got {e} back to queue {key}') r.rpush(key, msg) continue Pop a member from a list, if failed plop it back to the end. It's simple, explicit and just works™
- lbruder 6y ago
  It just works, until PUSHing the member back fails. At least use RPOPLPUSH and a separate worker queue to make sure you don't accidentally drop packets. Don't get me wrong, I'm using Redis queues myself and trying to get rid of the last remnants of RabbitMQ in our code, but there are use cases where RabbitMQ means less thinking about stuff, because it just works(tm)...
  - jlouis 6y ago
    The other classic variant here is when msg is poisonous. Then pushing it back onto the queue means it'll go again at some point, poisoning your system again. In practice, you need some dead-lettering with a message timeout. Something RabbitMQ does provide out of the box, for all its complexity.
- ayush--s 6y ago
  couple of problems with this: (1) RPUSH can fail (2) LPOP's result might never reach your client after redis has executed it.

Mandatum 6y ago

brew install rabbitmq? I agree rabbitmq is often adopted in places it shouldn't, or ill-configured.. But to completely get rid of it because you didn't take the time to RTFM when setting it up seems a little extreme. And redis/rabbitmq have completely different use-cases 80% of the time. Sounds like you were trying to get drunk on kombucha.

dragonsh 6y ago
You are right in our case redis failed spectacularly with celery queue when tasks in queue exceeded the memory of the machine. It took weeks to diagnose as error message were inconsistent and setting up a Message Queue cluster with redis is a nightmare. Went back again to tried and tested rabbitMQ and it works so well. Also adding new nodes and removing nodes so easy, just used ansible to setup Erlang cookie and connect the node (thanks to OTP and BEAM). The best part is for important task queue for which we cannot accept failure we built a mechanism for fault tolerance. When you work with high availability and fault tolerant queue rabbitMQ is so good, it can recover from hardware or even VM failures, can’t say the same for redis which was a nightmare even with redis cluster.

oauea 6y ago

Odd, I've been running it for years and it's been flawless. Integrating new clients is a breeze using their, or any amqp, library.

Lazare 6y ago

If you were capable of replacing your use of rabbitmq with 3 lines of python and a redis instance, you shouldn't have been using rabbitmq to start with. That doesn't mean redis is a drop in replacement for any of the valid uses for rabbitmq though...

lowwave 6y ago
can you share some of those feature where rabbitmq that is useful? Tks.

mister_hn 6y ago

With docker it takes 5 seconds to set a rabbitmq instance locally.

codeduck 6y ago

Amusingly, RabbitMQ is the one message broker that HASN'T given me grief in my career.

elisharobinson 6y ago

Your fault is using pika. I thought the same untill I tried some other python libraries . Try Oslo messaging . It's used in openstack (enough said).

st1ck 6y ago

Does your solution handle the situation when consumer crashes and queue has to be accumulated (while RAM allows) until consumer is up again (maybe 1 day later)? AFAIK ZMQ can't guarantee this.

twic 6y ago
It must be emphasised that, despite the name, ZeroMQ is not a message queue. It is a networking library. The current blurb says: > ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. And the old meme was "ZeroMQ is a replacement for Berkeley sockets". It's a pretty cool networking library. But it makes no sense to think of it in the same slot as RabbitMQ, or even Redis. That the GP mentioned it does make me wonder if they don't really understand what they're doing. EDIT I love that the official guide actually has this diagram in! Pieter Hintjens's death was a sad loss. http://zguide.zeromq.org/page:all#How-It-Began http://zguide.zeromq.org/page:all#How-It-Began
- jen20 6y ago
  As an aside, the ZeroMQ Guide is the finest piece of technical writing I have ever seen - closely followed by Pieter’s (sadly unfinished) “Scalable C” book. His non-technical writing is also excellent.
  - francislavoie 6y ago
    I had a fun time implementing the Paranoid Pirate pattern with my coworker a few years back. He wrote the server part in Python, I wrote the client in PHP. We essentially built it as a wrapper to run some C code our boss wrote that we didn't want to write a PHP extension for - we used Python as a broker to allow for some concurrency. Worked super well.
ayush--s 6y ago
a Redis list (what he's using) would handle this easily (I think Redis' PubSub also handles this)

ayush--s 6y ago

rabbitmq used to be one of the hardest things to self-host (even at relatively small scale). But it has been pretty stable since moving to Cloudamqp as a managed solution.

MaxBarraclough 6y ago
I had the same experience. I had particular trouble configuring encryption for connections. It didn't help that they'd switched configuration file-formats, but the documentation and tutorials still used the old format. Never had any stability issues though. Once it was up and running, it was solid.

smoe 6y ago

Sounds to me like the problem is, that the company taking out the cannon to shoot at sparrows and that nobody seem to have bothered to provide working dev environments? It feels common that people think they their every problem requires a planetary scale solution or one that handles every conceivable case that could occur before the heat death of the universe. Be that because your a startup and think you are going to need to support a hundred million concurrent users on launch day, or because your an enterprise and think because your are oh so important you need to use the same tools other important companies use. My own anecdote: When we refactored our early stage app we had a huge mess with the Redis based queue system. It was one of the biggest sources of errors and a massive pain to troubleshoot or even to monitor what is going on. So we investigated in a bunch of different solutions including all the usual contenders, and we ended up with: Let's just ditch the messages / queues altogether for now and just do boring old cron like jobs invoking internal api endpoints in regular intervals. This made 9 out of 10 cases a lot easier to maintain, and for the remainder, while temporary more difficult, we introduced a queue again at a much later stage. I'm not saying you shouldn't use and you can't benefit of RabbitMQ at small scales or any technology for that matter. But I think too often in tech decision making ones own or the companies perceived importance, what is cool or what would look good on a CV takes precedence over what really fits the problem in context.

kazinator 6y ago

ZeroMQ compiles to well over 600 kilobytes of machine code (just on 32 bit x86). $ size /usr/lib/i386-linux-gnu/libzmq.so text data bss dec hex filename 662134 12952 24 675110 a4d26 /usr/lib/i386-linux-gnu/libzmq.so Yikes, is there an operating system kernel with virtual memory, USB support, a few filesystems and a TCP/IP stack hiding in there? It's like 35% of glibc: $ size /lib/i386-linux-gnu/libc.so.6 text data bss dec hex filename 1917549 11624 11112 1940285 1d9b3d /lib/i386-linux-gnu/libc.so.6 Which is itself heavily bloated, trying to provide as much POSIX as it can.

mcsoft 6y ago

We used both zeromq and its sort-of-descendant nanomsg but eventually moved to NATS because of it's approach to high availability (every node knows the topology of the whole cluster) and zero management overhead. Some criticize NATS for the absence of message durability in the core technology, but we figured out we can drop this requirement in 95% of cases. Your microservices should be highly available anyway, so there's always a live consumer - and it's better to handle the message to it directly rather than introducing the overhead of storing a message somewhere and dealing with more-than-once delivery. There's a bit more to that like you should let your microservices exit gracefully and finish processing consumed messages. And of course in 5% of cases, where you can't allow to lose a single message, you have to use NATS Streaming or the likes, but so far we've been greatly impressed by NATS for high load.

nathan-io 6y ago

I'm equally surprised by your anecdote. I found it quite simple to install RabbitMQ server and its admin panel in my WSL local dev environment. And the cloud/prod instance took a few clicks (just spun up a DO Marketplace server image) followed by < five minutes of RabbitMQ user and firewall configuration. It was also dead simple to start using RabbitMQ within my application. I found a well maintained package, installed it, edited a couple lines of my application's config, and everything just worked. I specifically avoided Redis based on my understanding that it can't guarantee message persistence, so if it crashes, your unprocessed messages are lost.