4 ms·
In most cases, queues are going to be a lot faster than whatever they are feeding into. If you are doing something like sending every SQS message to a Lambda wi
by encoderer 6y ago
In most cases, queues are going to be a lot faster than whatever they are feeding into. If you are doing something like sending every SQS message to a Lambda with infinite concurrency then, sure, but many queues are feeding into a compute or persistence layer of some kind.
Back-pressure here would mean that emitters of messages into your queue behave differently when there is an excessive number of messages waiting to be processed.
For example:
- If it's a job queue, load-shed less important jobs until the queue is healthy.
- If you are ingesting a high volume of messages, be more aggressive with batching them together before inserting as a single queue message.
- sudhirj 6y agoAh, got it. I’m not entirely sure I want to put that in, though. The message I’m “preaching” is that you have full decoupling with queues - not sure that should be diluted. Wouldn’t this kind of checking defeat the purpose of using the queue in the first place? It seems unlikely that Amazon would stop selling stuff just because the order-confirmation-email-sending-service was backed up, for instance. The point of using SQS would be that the sales system wouldn’t have to care.
- encoderer 6y agoIn the universe of message queues, an email-per-sale queue is not going to be your problem. Think of the queue-as-message-bus pattern -- very common. You don't want to have a service with millions of pending messages to process, it forces you into painful, slow recovery, or making the tough choice of dropping everything. Anyway, great write up, and happy new year!