4 ms·

while this is a good overview, i think this article paints too rosy a picture of message queues, specifically > We can put as many messages as we want into the

by dastbe 6y ago

while this is a good overview, i think this article paints too rosy a picture of message queues, specifically > We can put as many messages as we want into the tube (let's assume we have a infinitely long tube) at whatever speed is comfortable to us. and > The receiver will never be impacted by our actions—they will pull out as many messages as they want at whatever rate is comfortable to them. and > Neither the sender nor the receiver are concerned with how the other works. First, I would argue that the sender and receiver still absolutely have to know how the other works, because your API contract is now the structure of all messages that enter the queue. You also need to be aware of the semantics of both client and receiver: if I receive a message I can't process, can I get rid of it? Do I need to retry it forever? And as a sender, will my receiver retry this message or do I need to track success? In synchronous systems you have the opportunity provide clearer feedback in these scenarios, which help mitigate or at least share responsibility in mitigating poison pill scenarios. It's also the case that when you introduce a queue you're adding an arbitrary buffer. During short transient outages this is usually ok, as your system has some slack somewhere at some time to drive down the pileup, but during persistent outages you can end up driving up a difficult to overcome buffer. And worse, in the event of time sensitive message a substantial amount of that buffer may be otherwise useless. Unless your receiver knows whether a message can be safely ignored, you can encounter outage scenarios that take hours or even days to recover from and exacerbate impact after recovery compared to if you just had no queue and shed load in the first place.

sudhirj 6y ago
Thanks for the points, yeah, definitely need to cover poison pills and dead letter queues. Re. size the main examples I'm looking at are SQS, Google's Pub/Sub etc which are pretty hard to fill up or overload. Will think about how to talk about recovery measures. The handling is very application specific, it seems that guidelines will boil down to 1) if your messages are important, suck it up and add more subscriber processing capacity or 2) if they're not important, purge the queue and restart your subscribers. Not sure I want to get into this kind level of advice without looking at the nature of the messages in the system.
- dastbe 6y ago
  > Re. size the main examples I'm looking at are SQS, Google's Pub/Sub etc which are pretty hard to fill up or overload. It's not the size of the queue supported by the queue service that's the problem, it is the size of the queue in relation to the receivers capacity to process it. The infinite buffer can become a liability when your receivers are unable to handle it. > 1) if your messages are important, suck it up and add more subscriber processing capacity or > > 2) if they're not important, purge the queue and restart your subscribers. But what if things aren't so binary? Say I have a queue of emails that have backed up because my email service (or one of its dependencies) had a multi hour outage on black friday. Some of these messages are time sensitive ("We're packing it!" for a delivery that already happened) and some must be sent ("Your account is being terminated, this is the last notification"). You would like to discard messages like the former (which are certainly the overwhelming bulk of your messages) while making sure that you absolutely send the latter. Oh, and it turns out your email service has a fixed rate limit on your account and they won't raise it. This stuff happens. Another example I've seen is a queue used as a workflow system: I enqueue commands (add X to Y, ensure X is in state Z, remove W from Y) in a backing system because, right, sometimes the dependencies for doing these operations are down for a little bit. But then one day they are down for a long-time, and the automation that has been set up on top of the external API has enqueued A LOT of commands. And a lot of those commands are now useless, things like an unprocessed "add X to Y" when there's a "remove X from Y" just a couple thousand messages later. My dependency is working to come back up to full capacity and isn't going to give me a higher rate limit, so I need to figure out how to drain this queue faster to get back to my normal line rate. Another way to put this is that queues introduce bimodal state. When the queue is empty or only experiences temporary periods of overfill, my service runs in one way and has predictable behavior. But when my queue becomes chronically overfilled, my service is in an entirely different state. This is certainly bad news for my system, but can also be bad news for adjacent parts of the greater system that expect certain behaviors out of my service.
  - sudhirj 6y ago
    Yeah... let me think about this. Not sure how to boil this down to generalized notes or advice.
kqr 6y ago
This was one of the things that made me look into lean manufacturing (book recommendation: The Machine That Changed the World) and lean product development (book recommendation: The Principles of Product Development Flow). The latter is more relevant for software systems, but the former was very good to have read first to see where software is different. These areas can teach us a lot about the dangers of queues and how to handle them carefully.

by dastbe 6y ago

sudhirj 6y ago

Thanks for the points, yeah, definitely need to cover poison pills and dead letter queues. Re. size the main examples I'm looking at are SQS, Google's Pub/Sub etc which are pretty hard to fill up or overload. Will think about how to talk about recovery measures. The handling is very application specific, it seems that guidelines will boil down to 1) if your messages are important, suck it up and add more subscriber processing capacity or 2) if they're not important, purge the queue and restart your subscribers. Not sure I want to get into this kind level of advice without looking at the nature of the messages in the system.

dastbe 6y ago
> Re. size the main examples I'm looking at are SQS, Google's Pub/Sub etc which are pretty hard to fill up or overload. It's not the size of the queue supported by the queue service that's the problem, it is the size of the queue in relation to the receivers capacity to process it. The infinite buffer can become a liability when your receivers are unable to handle it. > 1) if your messages are important, suck it up and add more subscriber processing capacity or > > 2) if they're not important, purge the queue and restart your subscribers. But what if things aren't so binary? Say I have a queue of emails that have backed up because my email service (or one of its dependencies) had a multi hour outage on black friday. Some of these messages are time sensitive ("We're packing it!" for a delivery that already happened) and some must be sent ("Your account is being terminated, this is the last notification"). You would like to discard messages like the former (which are certainly the overwhelming bulk of your messages) while making sure that you absolutely send the latter. Oh, and it turns out your email service has a fixed rate limit on your account and they won't raise it. This stuff happens. Another example I've seen is a queue used as a workflow system: I enqueue commands (add X to Y, ensure X is in state Z, remove W from Y) in a backing system because, right, sometimes the dependencies for doing these operations are down for a little bit. But then one day they are down for a long-time, and the automation that has been set up on top of the external API has enqueued A LOT of commands. And a lot of those commands are now useless, things like an unprocessed "add X to Y" when there's a "remove X from Y" just a couple thousand messages later. My dependency is working to come back up to full capacity and isn't going to give me a higher rate limit, so I need to figure out how to drain this queue faster to get back to my normal line rate. Another way to put this is that queues introduce bimodal state. When the queue is empty or only experiences temporary periods of overfill, my service runs in one way and has predictable behavior. But when my queue becomes chronically overfilled, my service is in an entirely different state. This is certainly bad news for my system, but can also be bad news for adjacent parts of the greater system that expect certain behaviors out of my service.
- sudhirj 6y ago
  Yeah... let me think about this. Not sure how to boil this down to generalized notes or advice.

kqr 6y ago

This was one of the things that made me look into lean manufacturing (book recommendation: The Machine That Changed the World) and lean product development (book recommendation: The Principles of Product Development Flow). The latter is more relevant for software systems, but the former was very good to have read first to see where software is different. These areas can teach us a lot about the dangers of queues and how to handle them carefully.