7 ms·

Message Queues: A Simple Guide with Analogies (2024)

111 points by byt3h3ad 9mo ago

coronapl 9mo ago
While queues definitely play an important role in microservices architecture, I think it’s worth clarifying that they’re not unique to it. A queue can fit perfectly in a monolith depending on the use case. I regularly use queues for handling critical operations that might require retrying, for having better visibility into failed jobs, ensuring FIFO guarantees, and more. Queues are such a useful tool for building any resilient architecture that framing them as primarily a microservices concern might cause unnecessary confusion.
- robertlagrant 9mo ago
  Totally agree. Banks use durable queues a lot to make sure things get processed. Or at least they used to.
  - coronapl 9mo ago
    The analogy could be: “Queues are the like the todos list of your team. The todo item (message) stays there until it is successfully completed. It can be handled by the producer (monolith) or it can be handled by someone else (microservices).”
- Aurornis 9mo ago
  Monoliths also have to scale to multiple servers eventually, so message queues are an important architectural component to understand regardless of the organization of your services.
  - charcircuit 9mo ago
    Even without multiple servers a single server itself has many cores. So if you aren't using multiple threads you are leaving performance on the table.
    - Aurornis 9mo ago
      A single multithreaded process usually doesn’t need an external message queue for sharing, though. I guess if you’re stuck with a single threaded language you would want a message queue though.
      - charcircuit 9mo ago
        No one specified external message queue. You can have a message queue within the process itself that can deliver messages from one thread to another. There are different kinds of messages queues for this purpose depending on how many threads will be producing messages to the message queue and how many threads will be consuming messages from it. While using message queues may not be neccessary for multithreading it is a very common setup. The ability to have a message queue to schedule work on a background thread is very common between many programs.
- NikolaNovak 9mo ago
  Absolutely, 100%. I work on PeopleSoft Enterprise Resource Planning applications - the "boring" back-office HR, Pay, Financials, Planning etc stuff. The core architecture is late 80s - mid 90s. Couple of big architectural changes when internet/browsers and then mobile really hit. But fundamentally it's a very legacy / old school application. Lots of COBOL, if that helps calibrate :-> We use queues pervasively. It's PeopleSoft's preferred integration method for other external applications, but over the years a large number of internal plumbing is now via queues as well. PeopleSoft Integration Broker is kind of like an internal proprietary ESB. So understanding queues and messaging is key to my PeopleSoft Administrator teams wherever I go (basically sysadmins in service of PeopleSoft application:).
  - coronapl 9mo ago
    Recently, I also started using queues for integrating with legacy health care applications. Most of them run on-promise and they don't have incoming internet connection for security reasons. The strategy is to send a message to a queue. The consumer application uses short polling to process the messages and then it can call a webhook to share the status of the job. Do you also follow a similar approach?
    - NikolaNovak 9mo ago
      If I understand it correctly, no; PeopleSoft is Legacy in some ways but it is actively developed and improved/maintained. The Peoplesoft Integration Broker is "modern-ish" from that perspective, and a proper middleware messaging system: https://docs.oracle.com/cd/E92519_02/pt856pbr3/eng/pt/tibr/concept_IntroductiontoPeopleSoftIntegrationBroker-076593.html?pli=ul_d74e34_tibr https://docs.oracle.com/cd/E92519_02/pt856pbr3/eng/pt/tibr/c... It'll do XML messages in somewhat proprietary format with other PeopleSoft applications, and "near-real-time" queues via web services with other applications in a fairly standardized way (WSDL etc). I think of PeopleSoft Integration Broker as a "mini, proprietary ESB", as inaccurate as it may be in details :).
prhn 9mo ago
This is surprisingly basic knowledge for ending up on the front page. It’s a good intro, but I’d love to read more about when to know it’s time to replace my synchronous inter service http requests with a queue. What metrics should I consider and what are the trade offs. I’ve learned some answers to this question over time, but these guys are theoretically message queue experts. I’d love to learn about more things to look out for. There are also different types of queues/exchanges and this is critical depending on the types of consumer or consumers you have. Should I use direct, fan out, etc? The next interesting question is when should I use a stream instead of a queue, which RabbitMQ also supports. My advice, having just migrated a set of message queues and streams from AWS(AvtiveMQ) to RabbitMQ is think long and hard before you add one. They become a black box of sorts and are way harder to debug than simple HTTP requests. Also, as others have pointed out, there are other important use cases for queues which come way before microservice comms. Async processing to free up servers is one. I’m surprised none of these were mentioned.
- Aurornis 9mo ago
  > This is surprisingly basic knowledge for ending up on the front page. Nothing wrong with that! Hacker News has a large audience of all skill levels. Well written explainers are always good to share, even for basic concepts.
  - coronapl 9mo ago
    Agree! In fact, I would appreciate more well written articles explaining basic concepts on the front page of Hacker News. It is always good to revisit some basic concepts, but it is even better to relearn them. I am surprised by how often I realize that my definition of a concept is wrong or just superficial.
    - SAI_Peregrinus 9mo ago
      Also it's nice to have a set of well-written explainers for when someone asks about a concept.
  - p1anecrazy 9mo ago
ImPleadThe5th 9mo ago
After spending most of my career hacking on these systems, I feel like queues very quickly become a hammer and every entity quickly becomes a nail. Just because you can keep two systems in complete sync doesn't mean you should. If you ever find yourself with more-or-less identical tables in two services you may have gone too far. Eventually you find yourself backfilling downstream services due to minor domain or business logic changes and scaling is a problem again.
emmanueloga_ 9mo ago
I’ve been thinking that defaulting to durable execution over lower‑level primitives like queues makes sense a lot of the time, what do you think? A lot of the "simple queue" use cases end up needing extra machinery like a transactional‑outbox pattern just to be reliable. Durable‑execution frameworks (DBOS/Temporal/etc.) give you retries, state, and consistency out of the box. Patterns like Sagas also tend to get stitched together on top of queues, but a DE workflow gives you the same guarantees with far less complexity. The main tradeoff I can think of is latency: DE engines add overhead, so for very high throughput, huge fan‑out, or ultra‑low‑latency pipelines, a bare‑bones queue + custom consumers might still be better. Curious where others draw the line between the two.
- jedberg 9mo ago
  Highly biased opinion here since I'm the CEO of DBOS: It'll be rare that the overhead actually has an effect, especially if you use a library like DBOS, which only adds a database write. You still have to write to and read from your queue, which is about as expensive as a database write/read.
- abelanger 9mo ago
  Drawing the boundary at high throughput, huge fan-out and ultra-low-latency is correct - I'd also add that MQs are often used for pub/sub and signaling. MQs are heavily optimized for reducing E2E latency between publishers and consumers in a way that DE engines are not, since DE engines usually rely on an ACID compliant database. Under load I've seen an order of magnitude difference in enqueue times (low single-digit milliseconds for the MQ p95 vs 10ms p95 for Postgres commit times). And AMQP has a number of routing features built-in (i.e. different exchange types) that you won't see in DE engines. Another way to think about it is that message queues usually provide an optional message durability layer alongside signaling and pub/sub. So if you need a very simple queue with retries _and_ you need pub/sub, I'd be eyeing an MQ (or a DE execution engine that supports basic pub/sub, like Hatchet). I wrote about our perspective on this here: https://hatchet.run/blog/durable-execution https://hatchet.run/blog/durable-execution ( disclaimer - I'm one of the people behind https://github.com/hatchet-dev/hatchet https://github.com/hatchet-dev/hatchet )
- 9mo ago
keithnz 9mo ago
this is not really a good guide to Message Queues, it's really only talking about them in context of one use of them. It doesn't really talk about the message queue at all and the basic differences between various message queue implementations. Your local AI chatbot is going to give you a much better overview, just take the title "Message Queues: A Simple Guide with Analogies" and it does a much better job

111 points by byt3h3ad 9mo ago

While queues definitely play an important role in microservices architecture, I think it’s worth clarifying that they’re not unique to it. A queue can fit perfectly in a monolith depending on the use case. I regularly use queues for handling critical operations that might require retrying, for having better visibility into failed jobs, ensuring FIFO guarantees, and more. Queues are such a useful tool for building any resilient architecture that framing them as primarily a microservices concern might cause unnecessary confusion.

robertlagrant 9mo ago
Totally agree. Banks use durable queues a lot to make sure things get processed. Or at least they used to.
- coronapl 9mo ago
  The analogy could be: “Queues are the like the todos list of your team. The todo item (message) stays there until it is successfully completed. It can be handled by the producer (monolith) or it can be handled by someone else (microservices).”
Aurornis 9mo ago
Monoliths also have to scale to multiple servers eventually, so message queues are an important architectural component to understand regardless of the organization of your services.
- charcircuit 9mo ago
  Even without multiple servers a single server itself has many cores. So if you aren't using multiple threads you are leaving performance on the table.
  - Aurornis 9mo ago
    A single multithreaded process usually doesn’t need an external message queue for sharing, though. I guess if you’re stuck with a single threaded language you would want a message queue though.
    - charcircuit 9mo ago
      No one specified external message queue. You can have a message queue within the process itself that can deliver messages from one thread to another. There are different kinds of messages queues for this purpose depending on how many threads will be producing messages to the message queue and how many threads will be consuming messages from it. While using message queues may not be neccessary for multithreading it is a very common setup. The ability to have a message queue to schedule work on a background thread is very common between many programs.
NikolaNovak 9mo ago
Absolutely, 100%. I work on PeopleSoft Enterprise Resource Planning applications - the "boring" back-office HR, Pay, Financials, Planning etc stuff. The core architecture is late 80s - mid 90s. Couple of big architectural changes when internet/browsers and then mobile really hit. But fundamentally it's a very legacy / old school application. Lots of COBOL, if that helps calibrate :-> We use queues pervasively. It's PeopleSoft's preferred integration method for other external applications, but over the years a large number of internal plumbing is now via queues as well. PeopleSoft Integration Broker is kind of like an internal proprietary ESB. So understanding queues and messaging is key to my PeopleSoft Administrator teams wherever I go (basically sysadmins in service of PeopleSoft application:).
- coronapl 9mo ago
  Recently, I also started using queues for integrating with legacy health care applications. Most of them run on-promise and they don't have incoming internet connection for security reasons. The strategy is to send a message to a queue. The consumer application uses short polling to process the messages and then it can call a webhook to share the status of the job. Do you also follow a similar approach?
  - NikolaNovak 9mo ago
    If I understand it correctly, no; PeopleSoft is Legacy in some ways but it is actively developed and improved/maintained. The Peoplesoft Integration Broker is "modern-ish" from that perspective, and a proper middleware messaging system: https://docs.oracle.com/cd/E92519_02/pt856pbr3/eng/pt/tibr/concept_IntroductiontoPeopleSoftIntegrationBroker-076593.html?pli=ul_d74e34_tibr https://docs.oracle.com/cd/E92519_02/pt856pbr3/eng/pt/tibr/c... It'll do XML messages in somewhat proprietary format with other PeopleSoft applications, and "near-real-time" queues via web services with other applications in a fairly standardized way (WSDL etc). I think of PeopleSoft Integration Broker as a "mini, proprietary ESB", as inaccurate as it may be in details :).

prhn 9mo ago

This is surprisingly basic knowledge for ending up on the front page. It’s a good intro, but I’d love to read more about when to know it’s time to replace my synchronous inter service http requests with a queue. What metrics should I consider and what are the trade offs. I’ve learned some answers to this question over time, but these guys are theoretically message queue experts. I’d love to learn about more things to look out for. There are also different types of queues/exchanges and this is critical depending on the types of consumer or consumers you have. Should I use direct, fan out, etc? The next interesting question is when should I use a stream instead of a queue, which RabbitMQ also supports. My advice, having just migrated a set of message queues and streams from AWS(AvtiveMQ) to RabbitMQ is think long and hard before you add one. They become a black box of sorts and are way harder to debug than simple HTTP requests. Also, as others have pointed out, there are other important use cases for queues which come way before microservice comms. Async processing to free up servers is one. I’m surprised none of these were mentioned.

Aurornis 9mo ago
> This is surprisingly basic knowledge for ending up on the front page. Nothing wrong with that! Hacker News has a large audience of all skill levels. Well written explainers are always good to share, even for basic concepts.
- coronapl 9mo ago
  Agree! In fact, I would appreciate more well written articles explaining basic concepts on the front page of Hacker News. It is always good to revisit some basic concepts, but it is even better to relearn them. I am surprised by how often I realize that my definition of a concept is wrong or just superficial.
  - SAI_Peregrinus 9mo ago
    Also it's nice to have a set of well-written explainers for when someone asks about a concept.
- p1anecrazy 9mo ago

ImPleadThe5th 9mo ago

After spending most of my career hacking on these systems, I feel like queues very quickly become a hammer and every entity quickly becomes a nail. Just because you can keep two systems in complete sync doesn't mean you should. If you ever find yourself with more-or-less identical tables in two services you may have gone too far. Eventually you find yourself backfilling downstream services due to minor domain or business logic changes and scaling is a problem again.

emmanueloga_ 9mo ago

I’ve been thinking that defaulting to durable execution over lower‑level primitives like queues makes sense a lot of the time, what do you think? A lot of the "simple queue" use cases end up needing extra machinery like a transactional‑outbox pattern just to be reliable. Durable‑execution frameworks (DBOS/Temporal/etc.) give you retries, state, and consistency out of the box. Patterns like Sagas also tend to get stitched together on top of queues, but a DE workflow gives you the same guarantees with far less complexity. The main tradeoff I can think of is latency: DE engines add overhead, so for very high throughput, huge fan‑out, or ultra‑low‑latency pipelines, a bare‑bones queue + custom consumers might still be better. Curious where others draw the line between the two.

jedberg 9mo ago
Highly biased opinion here since I'm the CEO of DBOS: It'll be rare that the overhead actually has an effect, especially if you use a library like DBOS, which only adds a database write. You still have to write to and read from your queue, which is about as expensive as a database write/read.
abelanger 9mo ago
Drawing the boundary at high throughput, huge fan-out and ultra-low-latency is correct - I'd also add that MQs are often used for pub/sub and signaling. MQs are heavily optimized for reducing E2E latency between publishers and consumers in a way that DE engines are not, since DE engines usually rely on an ACID compliant database. Under load I've seen an order of magnitude difference in enqueue times (low single-digit milliseconds for the MQ p95 vs 10ms p95 for Postgres commit times). And AMQP has a number of routing features built-in (i.e. different exchange types) that you won't see in DE engines. Another way to think about it is that message queues usually provide an optional message durability layer alongside signaling and pub/sub. So if you need a very simple queue with retries _and_ you need pub/sub, I'd be eyeing an MQ (or a DE execution engine that supports basic pub/sub, like Hatchet). I wrote about our perspective on this here: https://hatchet.run/blog/durable-execution https://hatchet.run/blog/durable-execution ( disclaimer - I'm one of the people behind https://github.com/hatchet-dev/hatchet https://github.com/hatchet-dev/hatchet )
9mo ago

keithnz 9mo ago

this is not really a good guide to Message Queues, it's really only talking about them in context of one use of them. It doesn't really talk about the message queue at all and the basic differences between various message queue implementations. Your local AI chatbot is going to give you a much better overview, just take the title "Message Queues: A Simple Guide with Analogies" and it does a much better job