3 ms·

The processing of my job might have a legit Lambda timeout of 5 minutes, because occasional proper evaluations take that amount of time (even though most take m

The processing of my job might have a legit Lambda timeout of 5 minutes, because occasional proper evaluations take that amount of time (even though most take much less). In my opinion, the fact that the SQS connector can't keep hold of that message after receiving a 429 response (while keeping the visibility state updated, that is something we were doing ourselves prior to this migration), and process it right away after capacity has been freed up, is a flaw. There's a 5 minute waiting time for the retry in this scenario that needn't be there, even if the lack of it would force the SQS connector to be slightly (but just slightly) more stateful.

rantallion 4y ago
It sounds like you're missing an opportunity to handle the SQS visibility yourself instead of leaning on the default handling. If you interact with the SQS API from within your code (whether that's in your Lambda function or some other orchestration like a Step Function), the default handling ceases and you're in full control. Do this and you can now decide exactly and programmatically when a message is made visible again. While it's not the focus of his talk, there's a good example of SQS message visibility done properly here: https://www.youtube.com/watch?v=I85MPC_5yXE&t=2614s https://www.youtube.com/watch?v=I85MPC_5yXE&t=2614s