11 ms·

How to use RabbitMQ in service integration

63 points by olikas 5y ago

100011_100001 5y ago
I find that one of the major drawbacks of RabbitMQ is how much CPU it requires to run efficiently. If you add clustering on top of it it can easily become the biggest CPU hog in a containerized cluster. However it does what it says, and it does quite well.
- amirkdv 5y ago
  Curious: what kind of load are you throwing at RabbitMQ?
nesarkvechnep 5y ago
When I proposed to use RabbitMQ for service integration my colleagues didn't agree to go with it because it "won't scale" and AWS doesn't provide a managed service. Because of an AWS fetish and awful development experience the product won't launch anytime soon.
- pyepye 5y ago
  AWS released the option to have a managed RabbitMQ broker under Amazon MQ back in November (2020) https://aws.amazon.com/about-aws/whats-new/2020/11/announcing-amazon-mq-rabbitmq/ https://aws.amazon.com/about-aws/whats-new/2020/11/announcin...
  - jpalomaki 5y ago
    One thing to note is the pricing. Clustering is not supported with the micro instance, so if you need it, you must go with at least mq.m5.large [1] and you need three of them, so the total cost is around $620 per month (or closer to $700 for example in eu-central-1). [1] https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-instance-types.html#rabbitmq-broker-instance-types https://docs.aws.amazon.com/amazon-mq/latest/developer-guide...
- cratermoon 5y ago
  At a previous employer I had what I thought was close to a slam dunk case for messaging with RabbitMQ. While I'm not the most experienced person in developing distributed messaging systems, and I expected technical objections, the disheartening response, and ultimate reason for rejection, was simply organizational inertia. In short, "you can't use RabbitMQ and messaging for that because we don't use messaging here". I'm sure there was some history that nobody mentioned or some lost institutional knowledge. The "grandma's ham" of systems design.
  - zomglings 5y ago
    I can sympathize with your colleagues. Developing the distributed messaging system is not where the bulk of the cost of running such a system goes - it's the maintenance and debugging that soak up time, money, and tears. While my past experiences with RabbitMQ in production have been stellar, I can see why a team would be hesitant to add this complexity to their infrastructure.
    - cratermoon 5y ago
      Oh here's the thing though: the company was already using RabbitMQ, as well as Kafka (for other things, mostly data science), and it was all deployed in Kubernetes and replicated across three data centers, including one in Amsterdam. The complexity was already there, but only for a very restricted set of applications.
    - datavirtue 5y ago
      All of these problems sound like they come from developers who don't know how to develop distributed systems. If anything this should be simplifying an otherwise intractible problem. I have seen people overuse messaging and had to fix what I could. I otherwise reach for RMQ as a simple solution to horizontal scaling and I write the software to easily switch brokers (Aws SQS, on prem RMQ, cloud RMQ). It's not something that you choose to adopt and then force developers to make every interaction a distributed message.
      - cratermoon 5y ago
- KptMarchewa 5y ago
  Why not use SQS if you're bound to AWS requirement? I've found RabbitMQ difficult operationally, and full of footguns (that can make you lose data), so I'm not sure why would you want to use it if you don't already have to.
  - dsign 5y ago
    Not SQS, but Google PubSub: everything was working allright, until one day messages started arriving with horrible latency and vital parts of our service went down. We spent close to two weeks debugging our infrastructure, because _sure it could not be PubSub_. Well, _it was_. I guess nothing too big, a small bit or tweak that changed in their service that didn't propagate to the client library we were using, or god knows what. Now we run RabbitMQ for everything; at least we control our versions and we can confidently blame ourselves when something happens. But RabbitMQ, standalone and clustered, has served us well across different systems for about fifteen years.
jpgvm 5y ago
RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do with Kafka but this had serious deficiencies when implementing queues because of the cumulative ack only nature of Kafka. Recently I have started using Pulsar which provides selective ack and all the best parts of AMQP without the complexity and unneeded parts. i.e it has things like scheduled delivery and TTLs in addition to the all important shared subscription which makes queues "just work" on top of streams. If you want something like RabbitMQ but with a simpler API and are comfortable with JVM services give Pulsar a go. It's not for everyone but if you are already using a lot of the big data stack it's probably a good fit.
- bironran 5y ago
  Can only echo parent. 3 places of work in varying sizes, 4 projects in varying maturity, not a single RMQ administration staff that was competent enough to reliably run the cluster. Which of course leads me to believe the problem isn't with the people but with the ridiculously high threshold of knowledge, experience and app developer self-control needed to run RMQ successfully. As parent said, many meltdowns later, I'm now firmly in the "No Rabbit!" camp. Redis pubsub/queues for immediate lossy delivery, kafka / gcp pubsub / aws sqs for less latency sensitive flows that require more consistency guarantees.
  - datavirtue 5y ago
    I concure. I like RMQ because I know how to configure and administer it but I would never trust anyone with it. My first exposure to it was on a project where the lead architect failed to read the documentation let alone understand any of it. Several years later I was able to fix it and then left it in the hands of some other incompetents.
- jayd16 5y ago
  Hmm, what issues did you run into? I've used it on a few projects in a mirrored way and it was always fine. Is the clustering the issue?
pkb 5y ago
DON'T. RabbitMQ is fragile and breaks under low memory/high swap conditions. Usually due to epmd breaking. I know, have to fix it (usually by killing epmd) daily.
- haswell 5y ago
  Is this a reason to not use it at all, or a reason to ensure the environment is properly tuned if you do use it? Seems your conclusion “DON’T” implies the former, but this seems unnecessarily extreme.
- winrid 5y ago
  I've seen downstream services fail and RMQ have to swap hundreds of millions of messages to disk, and then resume by itself as consumers picked up... maybe this issue you see affects certain versions?
eternalban 5y ago
I've skimmed the comments here and of course the article. Some points based on experience with Kafka, RabbitMQ and exploratory (and ahead of hype cycle) look at Pulsar: RabbitMQ is an excellent messaging middleware. But simply remember that it is not designed for optimal performance when holding on to data. "It is a river, not a lake". Performance is very sensitive to the amount of data that is in flight between send and receive end points. Kafka is a "lake", but will not give you the rich routing and diverse semantics of Rabbit. If you are building 'event sourced' systems, Kafka and similar systems are a better choice. Pulsar has a highly articulated architecture. It is built on Bookkeeper and decouples the persistent store servers from the client servicing servers. If you want to avoid the rebalancing pain of Kafka, Pulsar is the solution. However, Puslar has many more moving pieces. Both RabbitMQ and Pulsar are authentic 'middle ware', and extensible. Kafka is, true to its genesis, a highly performant distributed log. Durable, recoverable, and performant distributed messaging/journaling is inherently complex. Make sure you know what is it that you precisely require, and one of the above solutions will likely serve you well.
- rad_gruchalski 5y ago
  Mostly agreed but I’d say: Pulsar is a lake, Kafka is a river, RabbitMQ is a creek. Kafka still moves fast and once you out of retention, difficult to read past. Pulsar, thanks to adding bookies to the storage pool, can grow and does not have a retention to commit to.
perlgeek 5y ago
At work, we've been using a RabbitMQ instance as a central integration point (for about 7ish years now, I believe), and it's been working very well so far. My observations so far: * Running a single RabbitMQ is pretty boring in the good sense. * We haven't managed to switch to a cluster for HA yet; it seems that software that deals with RabbitMQ clusters must be cluster aware (consume from queues on all instances and the likes), and it wasn't worth our effort to fix all the applications. * In the long run, the lack of tooling is hurting us. Want to do green-blue deployments? Canary deployments? When your services run on HTTP(S), there are simply tools for that. When your services consume from AMQP queues? You have to go searching for solutions, possibly build your own plumbing. In the end, it turns out that we use publish/subscribe far less often that direct request/response patterns, so for new stuff I'd likely go with HTTPS instead of AMQP today.
mianos 5y ago
Same here in regards to liking RabbitMQ until you need to scale it. I wasted a huge amount of time years ago trying to build a HA cluster that worked when you hit it with a load. What didn't help was the pika python client had so many issues. One of my github problem reports had a 20 line demo showing how it broke with anything but a trivial load. I gave up soon after as I had further problems in other languages. Over a year later someone looked at it and said 'yep fails under load' and fixed it. One of those projects that showed so much promise only to make me sad.

63 points by olikas 5y ago

I find that one of the major drawbacks of RabbitMQ is how much CPU it requires to run efficiently. If you add clustering on top of it it can easily become the biggest CPU hog in a containerized cluster. However it does what it says, and it does quite well.

amirkdv 5y ago
Curious: what kind of load are you throwing at RabbitMQ?

nesarkvechnep 5y ago

When I proposed to use RabbitMQ for service integration my colleagues didn't agree to go with it because it "won't scale" and AWS doesn't provide a managed service. Because of an AWS fetish and awful development experience the product won't launch anytime soon.

pyepye 5y ago
AWS released the option to have a managed RabbitMQ broker under Amazon MQ back in November (2020) https://aws.amazon.com/about-aws/whats-new/2020/11/announcing-amazon-mq-rabbitmq/ https://aws.amazon.com/about-aws/whats-new/2020/11/announcin...
- jpalomaki 5y ago
  One thing to note is the pricing. Clustering is not supported with the micro instance, so if you need it, you must go with at least mq.m5.large [1] and you need three of them, so the total cost is around $620 per month (or closer to $700 for example in eu-central-1). [1] https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-instance-types.html#rabbitmq-broker-instance-types https://docs.aws.amazon.com/amazon-mq/latest/developer-guide...
cratermoon 5y ago
At a previous employer I had what I thought was close to a slam dunk case for messaging with RabbitMQ. While I'm not the most experienced person in developing distributed messaging systems, and I expected technical objections, the disheartening response, and ultimate reason for rejection, was simply organizational inertia. In short, "you can't use RabbitMQ and messaging for that because we don't use messaging here". I'm sure there was some history that nobody mentioned or some lost institutional knowledge. The "grandma's ham" of systems design.
- zomglings 5y ago
  I can sympathize with your colleagues. Developing the distributed messaging system is not where the bulk of the cost of running such a system goes - it's the maintenance and debugging that soak up time, money, and tears. While my past experiences with RabbitMQ in production have been stellar, I can see why a team would be hesitant to add this complexity to their infrastructure.
  - cratermoon 5y ago
    Oh here's the thing though: the company was already using RabbitMQ, as well as Kafka (for other things, mostly data science), and it was all deployed in Kubernetes and replicated across three data centers, including one in Amsterdam. The complexity was already there, but only for a very restricted set of applications.
  - datavirtue 5y ago
    All of these problems sound like they come from developers who don't know how to develop distributed systems. If anything this should be simplifying an otherwise intractible problem. I have seen people overuse messaging and had to fix what I could. I otherwise reach for RMQ as a simple solution to horizontal scaling and I write the software to easily switch brokers (Aws SQS, on prem RMQ, cloud RMQ). It's not something that you choose to adopt and then force developers to make every interaction a distributed message.
    - cratermoon 5y ago
KptMarchewa 5y ago
Why not use SQS if you're bound to AWS requirement? I've found RabbitMQ difficult operationally, and full of footguns (that can make you lose data), so I'm not sure why would you want to use it if you don't already have to.
- dsign 5y ago
  Not SQS, but Google PubSub: everything was working allright, until one day messages started arriving with horrible latency and vital parts of our service went down. We spent close to two weeks debugging our infrastructure, because _sure it could not be PubSub_. Well, _it was_. I guess nothing too big, a small bit or tweak that changed in their service that didn't propagate to the client library we were using, or god knows what. Now we run RabbitMQ for everything; at least we control our versions and we can confidently blame ourselves when something happens. But RabbitMQ, standalone and clustered, has served us well across different systems for about fifteen years.

jpgvm 5y ago

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do with Kafka but this had serious deficiencies when implementing queues because of the cumulative ack only nature of Kafka. Recently I have started using Pulsar which provides selective ack and all the best parts of AMQP without the complexity and unneeded parts. i.e it has things like scheduled delivery and TTLs in addition to the all important shared subscription which makes queues "just work" on top of streams. If you want something like RabbitMQ but with a simpler API and are comfortable with JVM services give Pulsar a go. It's not for everyone but if you are already using a lot of the big data stack it's probably a good fit.

bironran 5y ago
Can only echo parent. 3 places of work in varying sizes, 4 projects in varying maturity, not a single RMQ administration staff that was competent enough to reliably run the cluster. Which of course leads me to believe the problem isn't with the people but with the ridiculously high threshold of knowledge, experience and app developer self-control needed to run RMQ successfully. As parent said, many meltdowns later, I'm now firmly in the "No Rabbit!" camp. Redis pubsub/queues for immediate lossy delivery, kafka / gcp pubsub / aws sqs for less latency sensitive flows that require more consistency guarantees.
- datavirtue 5y ago
  I concure. I like RMQ because I know how to configure and administer it but I would never trust anyone with it. My first exposure to it was on a project where the lead architect failed to read the documentation let alone understand any of it. Several years later I was able to fix it and then left it in the hands of some other incompetents.
jayd16 5y ago
Hmm, what issues did you run into? I've used it on a few projects in a mirrored way and it was always fine. Is the clustering the issue?

pkb 5y ago

DON'T. RabbitMQ is fragile and breaks under low memory/high swap conditions. Usually due to epmd breaking. I know, have to fix it (usually by killing epmd) daily.

haswell 5y ago
Is this a reason to not use it at all, or a reason to ensure the environment is properly tuned if you do use it? Seems your conclusion “DON’T” implies the former, but this seems unnecessarily extreme.
winrid 5y ago
I've seen downstream services fail and RMQ have to swap hundreds of millions of messages to disk, and then resume by itself as consumers picked up... maybe this issue you see affects certain versions?

eternalban 5y ago

I've skimmed the comments here and of course the article. Some points based on experience with Kafka, RabbitMQ and exploratory (and ahead of hype cycle) look at Pulsar: RabbitMQ is an excellent messaging middleware. But simply remember that it is not designed for optimal performance when holding on to data. "It is a river, not a lake". Performance is very sensitive to the amount of data that is in flight between send and receive end points. Kafka is a "lake", but will not give you the rich routing and diverse semantics of Rabbit. If you are building 'event sourced' systems, Kafka and similar systems are a better choice. Pulsar has a highly articulated architecture. It is built on Bookkeeper and decouples the persistent store servers from the client servicing servers. If you want to avoid the rebalancing pain of Kafka, Pulsar is the solution. However, Puslar has many more moving pieces. Both RabbitMQ and Pulsar are authentic 'middle ware', and extensible. Kafka is, true to its genesis, a highly performant distributed log. Durable, recoverable, and performant distributed messaging/journaling is inherently complex. Make sure you know what is it that you precisely require, and one of the above solutions will likely serve you well.

rad_gruchalski 5y ago
Mostly agreed but I’d say: Pulsar is a lake, Kafka is a river, RabbitMQ is a creek. Kafka still moves fast and once you out of retention, difficult to read past. Pulsar, thanks to adding bookies to the storage pool, can grow and does not have a retention to commit to.

perlgeek 5y ago

At work, we've been using a RabbitMQ instance as a central integration point (for about 7ish years now, I believe), and it's been working very well so far. My observations so far: * Running a single RabbitMQ is pretty boring in the good sense. * We haven't managed to switch to a cluster for HA yet; it seems that software that deals with RabbitMQ clusters must be cluster aware (consume from queues on all instances and the likes), and it wasn't worth our effort to fix all the applications. * In the long run, the lack of tooling is hurting us. Want to do green-blue deployments? Canary deployments? When your services run on HTTP(S), there are simply tools for that. When your services consume from AMQP queues? You have to go searching for solutions, possibly build your own plumbing. In the end, it turns out that we use publish/subscribe far less often that direct request/response patterns, so for new stuff I'd likely go with HTTPS instead of AMQP today.

mianos 5y ago

Same here in regards to liking RabbitMQ until you need to scale it. I wasted a huge amount of time years ago trying to build a HA cluster that worked when you hit it with a load. What didn't help was the pika python client had so many issues. One of my github problem reports had a 20 line demo showing how it broke with anything but a trivial load. I gave up soon after as I had further problems in other languages. Over a year later someone looked at it and said 'yep fails under load' and fixed it. One of those projects that showed so much promise only to make me sad.