5 ms·

In enterprise environments, I more often see overuse of message brokers. I am all for targeted use of messaging systems, but what I usually see is an all or not

In enterprise environments, I more often see overuse of message brokers. I am all for targeted use of messaging systems, but what I usually see is an all or nothing approach where as soon as a team starts using messaging they use it for _all_ interprocess communication. This has some very painful downsides. First, traditional message brokers (ie queues, topics, persistence, etc) introduce a potential bottleneck for all communication, and on top of that they are _slow_ compared to just using the network (ie HTTP, with no middleman). I've had customers who start breaking apart their monoliths and replacing with microservices, and since the monolith used messaging they use messaging between microservices as well. Well, the brokers were already scaled to capacity for the old system, so going from "1" monolith service to "10" microservices means deploying 10x more messaging infrastructure. Traditional messaging brokers also don't scale all that well horizontally, so that 10x is probably more like 20x. Second, performance tuning becomes harder for people to understand. A core tenant of "lean" is to reduce the number/length of queues in a system to get more predictable latency and to make it easier to diagnose bottlenecks. But everyone always does the opposite with messaging: massive queues everywhere. By the time the alarms go off, multiple queues are backing up and no one is clear on where the actual bottleneck is. What I would like to see more of is _strategic_ use of messaging. For example, a work queue to scale out processing, but the workers use HTTP or some other synchronous method to call downstream services. Limiting messaging to very few points in the process still give a lot of the async benefits to the original client, while limiting how often requests hit the broker and making it easier to diagnose performance problems.

_pmf_ 7y ago
> First, traditional message brokers (ie queues, topics, persistence, etc) introduce a potential bottleneck for all communication, and on top of that they are _slow_ compared to just using the network (ie HTTP, with no middleman). How is multiplexing supposed to work with plain HTTP without middleman? One of the main uses for message queues is efficient fan-out (m + n connections instead of m x n).
lulf 7y ago
I recommend looking at a standard messaging protocol like AMQP 1.0, which will allow you to implement the request-response pattern in an efficient manner without message brokers. Either “direct” client server as with HTTP , or via an intermediary such as the Apache Qpid Dispatch Router. With the dispatch router, you can use pattern matching to specify if addresses should be routed to a broker or be routed directly to a particular service. This is way you can get the semantics that best fits your use case.
- geezerjay 7y ago
  > I recommend looking at a standard messaging protocol like AMQP 1.0, which will allow you to implement the request-response pattern in an efficient manner without message brokers. Honest question: if the goal is to implement the request/response pattern then why should anyone adopt AMQP instead of plain REST/RPC-over-HTTP?
  - lulf 7y ago
    If you don’t use any other pattern than request-response, I agree there is no point. If you have a mix of pub-sub, work queues and request-response, it could simplify your dependencies perhaps. Also AMQP 1.0 has some nice async capabilities and acknowledge modes that I believe goes beyond what http/2 and grpc supports today. OTOH I don’t have any real world experience operating such a mix of different communication patterns, so it could be the advantage is insignificant.
    - geezerjay 7y ago
      > If you have a mix of pub-sub, work queues and request-response, it could simplify your dependencies perhaps. That's a good point. Indeed if the project is already rolling with a message bus then it wouldn't make much sense to increase complexity just to use a specific message exchange pattern.
  - dagi3d 7y ago
    eg. That request might take more time than what a http request should. Also, by having workers that consume specific queues/topics, it's easier to scale them with more granularity
nizmow 7y ago
An intermediary message broker can give you a HUGE benefit though. A touch more robustness (a service can disappear for a short while and hopefully nobody even notices), you get load balancing "for free" (with carefully designed services), and if you're careful with your framework you can keep failed messages sitting around for you to analyse or even replay later. Most likely with HTTP you're going to need some kind of middle man anyway (load balancers, proxies, etc). Though I agree with your conclusion, and I think it mostly stands for almost all tools. Know your problems and apply the appropriate solutions.
jillesvangurp 7y ago
Yes. For simple setups, I'm usually biased towards having less moving parts. Queues sound nice until you start thinking about the edge cases with respect to errors, lost messages, rolling restarts, etc. In short, pretty much every system with queues I've ever seen this was a headache. There are good solutions for this of course but they require extra work like checking dead letter queues, devops related to monitoring of and managing the queues and depending systems, etc. If find lots of teams cut corners here and then have to deal with the inevitable fallout of things going wrong. An alternative that is usually simple to implement is some kind of feed based system that you poll periodically. This can be dynamic or even static files. Feeds may be cached. Depending systems can simply poll this. If they go down, they can catch up when they come back. If the feed system goes down, they will catch up when it comes back up. It also scales to large numbers of subscribers. You can do streaming or non streaming variants of this. You can even poll from a browser.
- kbouck 7y ago
  > Feeds may be cached. Depending systems can simply poll this. If they go down, they can catch up when they come back. This is how Kafka works. The "feed" of messages is stored as a log. Clients maintain a reference to their latest position in the log, and pick up where they left off upon reconnecting.
8fingerlouie 7y ago
I work for a company that uses message queues for just about everything that moves between (different) hardware, and we've yet to experience problems from a properly configured queue setup. We've used message queues for 30+ years, and I would say things like reliable messaging/guaranteed delivery, exactly once delivery, as well as not having to care about byte order, has saved us a lot of trouble over the years. We move data between mainframes, pc servers, and whatever system the customers use, and once the security setup is in place, the queues deliver data reliably. Without message queues, each application would have to know the details about the receiver, or each client application would have to know the details about the sender, and the implementation would be very specific. With message queues, you can replace part of your implementation, i.e. migrate it from the mainframe to a unix daemon, and nobody will notice. That being said, it does come at a heavy cost, and as i started out by saying, we only use it for X-platform messaging. Anything done locally communicates via "something else", files, pipes, shared buffers, or similar. Also, anything idempotent, if using queues at all, is not using "exactly once" delivery, which has a large performance overhead.
chapium 7y ago
> Second, performance tuning becomes harder for people to understand. A core tenant of "lean" is to reduce the number/length of queues in a system to get more predictable latency and to make it easier to diagnose bottlenecks. But everyone always does the opposite with messaging: massive queues everywhere. By the time the alarms go off, multiple queues are backing up and no one is clear on where the actual bottleneck is. This is avoidable however. Its trivial to aggregate queue depth across nodes and report it to an alerting service.
nine_k 7y ago
Queues and messages is a great way to structure application logic (see Erlang, Go, etc). But all message exchange need not be run through the same (heavyweight) central broker. In an app I recently worked on we used SNS for heavy lifting (e.g. it is replicated across regions), then a few smaller local queues, including just in-process, in-memory queues. It worked well. Though yes, you want throttling, backpressure, and good instrumentation.

_pmf_ 7y ago

> First, traditional message brokers (ie queues, topics, persistence, etc) introduce a potential bottleneck for all communication, and on top of that they are _slow_ compared to just using the network (ie HTTP, with no middleman). How is multiplexing supposed to work with plain HTTP without middleman? One of the main uses for message queues is efficient fan-out (m + n connections instead of m x n).

lulf 7y ago

I recommend looking at a standard messaging protocol like AMQP 1.0, which will allow you to implement the request-response pattern in an efficient manner without message brokers. Either “direct” client server as with HTTP , or via an intermediary such as the Apache Qpid Dispatch Router. With the dispatch router, you can use pattern matching to specify if addresses should be routed to a broker or be routed directly to a particular service. This is way you can get the semantics that best fits your use case.

geezerjay 7y ago
> I recommend looking at a standard messaging protocol like AMQP 1.0, which will allow you to implement the request-response pattern in an efficient manner without message brokers. Honest question: if the goal is to implement the request/response pattern then why should anyone adopt AMQP instead of plain REST/RPC-over-HTTP?
- lulf 7y ago
  If you don’t use any other pattern than request-response, I agree there is no point. If you have a mix of pub-sub, work queues and request-response, it could simplify your dependencies perhaps. Also AMQP 1.0 has some nice async capabilities and acknowledge modes that I believe goes beyond what http/2 and grpc supports today. OTOH I don’t have any real world experience operating such a mix of different communication patterns, so it could be the advantage is insignificant.
  - geezerjay 7y ago
    > If you have a mix of pub-sub, work queues and request-response, it could simplify your dependencies perhaps. That's a good point. Indeed if the project is already rolling with a message bus then it wouldn't make much sense to increase complexity just to use a specific message exchange pattern.
- dagi3d 7y ago
  eg. That request might take more time than what a http request should. Also, by having workers that consume specific queues/topics, it's easier to scale them with more granularity

nizmow 7y ago

An intermediary message broker can give you a HUGE benefit though. A touch more robustness (a service can disappear for a short while and hopefully nobody even notices), you get load balancing "for free" (with carefully designed services), and if you're careful with your framework you can keep failed messages sitting around for you to analyse or even replay later. Most likely with HTTP you're going to need some kind of middle man anyway (load balancers, proxies, etc). Though I agree with your conclusion, and I think it mostly stands for almost all tools. Know your problems and apply the appropriate solutions.

jillesvangurp 7y ago

Yes. For simple setups, I'm usually biased towards having less moving parts. Queues sound nice until you start thinking about the edge cases with respect to errors, lost messages, rolling restarts, etc. In short, pretty much every system with queues I've ever seen this was a headache. There are good solutions for this of course but they require extra work like checking dead letter queues, devops related to monitoring of and managing the queues and depending systems, etc. If find lots of teams cut corners here and then have to deal with the inevitable fallout of things going wrong. An alternative that is usually simple to implement is some kind of feed based system that you poll periodically. This can be dynamic or even static files. Feeds may be cached. Depending systems can simply poll this. If they go down, they can catch up when they come back. If the feed system goes down, they will catch up when it comes back up. It also scales to large numbers of subscribers. You can do streaming or non streaming variants of this. You can even poll from a browser.

kbouck 7y ago
> Feeds may be cached. Depending systems can simply poll this. If they go down, they can catch up when they come back. This is how Kafka works. The "feed" of messages is stored as a log. Clients maintain a reference to their latest position in the log, and pick up where they left off upon reconnecting.

8fingerlouie 7y ago

I work for a company that uses message queues for just about everything that moves between (different) hardware, and we've yet to experience problems from a properly configured queue setup. We've used message queues for 30+ years, and I would say things like reliable messaging/guaranteed delivery, exactly once delivery, as well as not having to care about byte order, has saved us a lot of trouble over the years. We move data between mainframes, pc servers, and whatever system the customers use, and once the security setup is in place, the queues deliver data reliably. Without message queues, each application would have to know the details about the receiver, or each client application would have to know the details about the sender, and the implementation would be very specific. With message queues, you can replace part of your implementation, i.e. migrate it from the mainframe to a unix daemon, and nobody will notice. That being said, it does come at a heavy cost, and as i started out by saying, we only use it for X-platform messaging. Anything done locally communicates via "something else", files, pipes, shared buffers, or similar. Also, anything idempotent, if using queues at all, is not using "exactly once" delivery, which has a large performance overhead.

chapium 7y ago

> Second, performance tuning becomes harder for people to understand. A core tenant of "lean" is to reduce the number/length of queues in a system to get more predictable latency and to make it easier to diagnose bottlenecks. But everyone always does the opposite with messaging: massive queues everywhere. By the time the alarms go off, multiple queues are backing up and no one is clear on where the actual bottleneck is. This is avoidable however. Its trivial to aggregate queue depth across nodes and report it to an alerting service.

nine_k 7y ago

Queues and messages is a great way to structure application logic (see Erlang, Go, etc). But all message exchange need not be run through the same (heavyweight) central broker. In an app I recently worked on we used SNS for heavy lifting (e.g. it is replicated across regions), then a few smaller local queues, including just in-process, in-memory queues. It worked well. Though yes, you want throttling, backpressure, and good instrumentation.