7 ms·

RabbitMQ is great. One of the few pieces of software I've used that "just works". The only downside is once you get message-queue-pilled, you start seeing oppo

by akyu 6y ago

RabbitMQ is great. One of the few pieces of software I've used that "just works". The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.

nailer 6y ago
+1. Discovered RabbitMQ/AMQP around 2010, since then tech went through a 2015-era wave of HTTP microservices that has come, and, largely gone or moved to MQ.
- fluxsauce 6y ago
  moved to MQ Are you referring to IBM MQ?
  - jonesetc 6y ago
    probably just meant message queues in general.
- carterklein13 6y ago
  When you say "gone or moved to MQ" - if not moved to messaging services like RabbitMQ/NATS/etc, where else could things have gone? At least from my experience, HTTP microservices are still very common, especially when using things like AWS Lambdas. I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc, or more and more I see lightweight systems going completely serverless and just using lambdas - which are HTTP microservices.
  - nailer 6y ago
    > When you say "gone or moved to MQ" - if not moved to messaging services like RabbitMQ/NATS/etc, where else could things have gone? They could have stayed trying to do continually running microservices on HTTP. > I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc I do too. > more and more I see lightweight systems going completely serverless and just using lambdas - which are HTTP microservices. Likewise. But long running HTTP microservices are lame, and everybody realises that now, despite it being a cool idea back in 2015.
    - carterklein13 6y ago
      To be fair, I started working post-2015, so I've actually never come face-to-face with a long running HTTP microservice backend... what would something like that even look like? I'm thinking of systems I've worked on that use a messaging queue, but that only rely on HTTP requests - is that what it would be? So like, I'd make a request to a microservice behind an endpoint, which in turn would make requests to 3 more microservices behind other endpoints? If so, I'm certainly glad that idea isn't cool anymore because that seems greatly inefficient :)
dkersten 6y ago
How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that? Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers? My main use-case is to receive status/change notifications from a service running elsewhere from the API server servicing the UI, in order to avoid polling for new data.
- zonk_ 6y ago
  We use it in a fairly big scale for our slack bot system. It was just set up once, as akyu said, and since then it just works. Whenever we had troubles, it was always anything other than RabbitMQ. I've also looked into other solutions (ActiveMQ, Google PubSub, ...) and RabbitMQ is by far the most straight-forward and quick to set up. There are some edge cases that it doesn't cover as well, for example automatic retries, but there are some "RabbitMQ patterns" to make it work. For a simple message broker/queue system, it's great and the docs are also great.
  - dkersten 6y ago
    Was it easy to setup in terms of reliability and failover? Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising! Is there anything I should keep in mind for running it in production? Any best practices or gotchas, based on your experience (eg don't run in docker, or make sure there's lots of RAM or things like that)? I guess its all in the production checklist. I need to read through it all again!
    - ekimekim 6y ago
      This came up in several other threads here: Don't use RabbitMQ's clustering. It's surprisingly brittle and hard to recover from. The accepted wisdom that I've seen is to run a single broker with a completely independent hot spare. But of course switching over to your hot spare will violate most of the guarentees that Rabbit gives you around durability, ordering etc, so you have to be very careful how you use it. I desperately want to like Rabbit (and have used it heavily in the past) but right now I wouldn't use it if I can get away with anything else, it just has no real HA story.
MR4D 6y ago
> The only downside is once you get message-queue-pilled, ... I think this is why email will never die. It's basically turned into a huge message queue. Even voice mails come into my inbox. ====== EDIT - I meant to say "huge universal message queue" and left out the word "universal" accidentally
- pjc50 6y ago
  It always was a message queue in a very literal sense. There's a lot of work in mailer-daemons to ensure that email has as reliable as possible delivery in a store-and-forward system..
  - MR4D 6y ago
    You're correct - I left out the word "universal" accidentally, which would have made my intent much more clear. Thanks for catching that.
- guptaneil 6y ago
  I think this is what a lot of people who complain about Slack don't get. It's just a better message queue for your business. The fact that you can funnel all your business events, regardless of whether they originate from humans or bots, into one place and then each worker (again, either human or bot) can subscribe/filter/react to relevant events is super powerful. However, if you try to use it as a corporate SMS platform or email replacement, you will very quickly feel overwhelmed because both of those message queues are designed for much lower throughput.
- emmelaich 6y ago
  And you can literally use the maildir format for a queue! https://pypi.org/project/dirq/ https://pypi.org/project/dirq/ Perl had the original implementation, and there are implementations in other languages.
  - MR4D 6y ago
    I forgot about that - used to be a cool hack!

by akyu 6y ago

nailer 6y ago

+1. Discovered RabbitMQ/AMQP around 2010, since then tech went through a 2015-era wave of HTTP microservices that has come, and, largely gone or moved to MQ.

fluxsauce 6y ago
moved to MQ Are you referring to IBM MQ?
- jonesetc 6y ago
  probably just meant message queues in general.
carterklein13 6y ago
When you say "gone or moved to MQ" - if not moved to messaging services like RabbitMQ/NATS/etc, where else could things have gone? At least from my experience, HTTP microservices are still very common, especially when using things like AWS Lambdas. I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc, or more and more I see lightweight systems going completely serverless and just using lambdas - which are HTTP microservices.
- nailer 6y ago
  > When you say "gone or moved to MQ" - if not moved to messaging services like RabbitMQ/NATS/etc, where else could things have gone? They could have stayed trying to do continually running microservices on HTTP. > I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc I do too. > more and more I see lightweight systems going completely serverless and just using lambdas - which are HTTP microservices. Likewise. But long running HTTP microservices are lame, and everybody realises that now, despite it being a cool idea back in 2015.
  - carterklein13 6y ago
    To be fair, I started working post-2015, so I've actually never come face-to-face with a long running HTTP microservice backend... what would something like that even look like? I'm thinking of systems I've worked on that use a messaging queue, but that only rely on HTTP requests - is that what it would be? So like, I'd make a request to a microservice behind an endpoint, which in turn would make requests to 3 more microservices behind other endpoints? If so, I'm certainly glad that idea isn't cool anymore because that seems greatly inefficient :)

dkersten 6y ago

How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that? Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers? My main use-case is to receive status/change notifications from a service running elsewhere from the API server servicing the UI, in order to avoid polling for new data.

zonk_ 6y ago
We use it in a fairly big scale for our slack bot system. It was just set up once, as akyu said, and since then it just works. Whenever we had troubles, it was always anything other than RabbitMQ. I've also looked into other solutions (ActiveMQ, Google PubSub, ...) and RabbitMQ is by far the most straight-forward and quick to set up. There are some edge cases that it doesn't cover as well, for example automatic retries, but there are some "RabbitMQ patterns" to make it work. For a simple message broker/queue system, it's great and the docs are also great.
- dkersten 6y ago
  Was it easy to setup in terms of reliability and failover? Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising! Is there anything I should keep in mind for running it in production? Any best practices or gotchas, based on your experience (eg don't run in docker, or make sure there's lots of RAM or things like that)? I guess its all in the production checklist. I need to read through it all again!
  - ekimekim 6y ago
    This came up in several other threads here: Don't use RabbitMQ's clustering. It's surprisingly brittle and hard to recover from. The accepted wisdom that I've seen is to run a single broker with a completely independent hot spare. But of course switching over to your hot spare will violate most of the guarentees that Rabbit gives you around durability, ordering etc, so you have to be very careful how you use it. I desperately want to like Rabbit (and have used it heavily in the past) but right now I wouldn't use it if I can get away with anything else, it just has no real HA story.

MR4D 6y ago

> The only downside is once you get message-queue-pilled, ... I think this is why email will never die. It's basically turned into a huge message queue. Even voice mails come into my inbox. ====== EDIT - I meant to say "huge universal message queue" and left out the word "universal" accidentally

pjc50 6y ago
It always was a message queue in a very literal sense. There's a lot of work in mailer-daemons to ensure that email has as reliable as possible delivery in a store-and-forward system..
- MR4D 6y ago
  You're correct - I left out the word "universal" accidentally, which would have made my intent much more clear. Thanks for catching that.
guptaneil 6y ago
I think this is what a lot of people who complain about Slack don't get. It's just a better message queue for your business. The fact that you can funnel all your business events, regardless of whether they originate from humans or bots, into one place and then each worker (again, either human or bot) can subscribe/filter/react to relevant events is super powerful. However, if you try to use it as a corporate SMS platform or email replacement, you will very quickly feel overwhelmed because both of those message queues are designed for much lower throughput.
emmelaich 6y ago
And you can literally use the maildir format for a queue! https://pypi.org/project/dirq/ https://pypi.org/project/dirq/ Perl had the original implementation, and there are implementations in other languages.
- MR4D 6y ago
  I forgot about that - used to be a cool hack!