4 ms·

Ooh boy. Undocumented limitations caused major problems with us with AWS SQS. It is now in the documentation (maybe because I complained to our AWS rep), but S

Ooh boy. Undocumented limitations caused major problems with us with AWS SQS. It is now in the documentation (maybe because I complained to our AWS rep), but SQS has pool of 20k messages that it will pull from when serving requests. If you are using a FIFO queue and you have 20k messages with the same message group ID in that buffer, then you are unable to process any other message regardless of how large the queue is. It caused multiple severe degradations until we rearchitected how messages are grouped. If only it had been in the documentation from day 1. Gotta love it!

HeavyStorm 6y ago
Wow, that's a scary story.
- jugg1es 6y ago
  Yea, we had a queue backlog of 3 million (that were bound by an SLA) by the time I figured out what was going on.
shaicoleman 6y ago
Pretty scary that it just breaks without any error messages, and without an easy way to clean up the damage. Queue overflows are not uncommon. "For FIFO queues, there can be a maximum of 20,000 inflight messages (received from a queue by a consumer, but not yet deleted from the queue). If you reach this quota, Amazon SQS returns no error messages. If your queue has a large backlog of 20,000 or more messages with the same message group ID, FIFO queues might be unable to return the messages that have a different message group ID but were sent to the queue at a later time until you successfully consume the messages from the backlog." Link to the documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
- jugg1es 6y ago
  I only figured out what was happening by trial and error while trying to fix the production issue where I was moving 1000 messages at a time into a 2nd queue. Once I had dequeued 20k, everything starting pumping like normal. That's when I fired off an email to our AWS rep asking for confirmation from the SQS dev team. Within a couple weeks, the docs you just quoted were added.

HeavyStorm 6y ago

Wow, that's a scary story.

jugg1es 6y ago
Yea, we had a queue backlog of 3 million (that were bound by an SLA) by the time I figured out what was going on.

shaicoleman 6y ago

Pretty scary that it just breaks without any error messages, and without an easy way to clean up the damage. Queue overflows are not uncommon. "For FIFO queues, there can be a maximum of 20,000 inflight messages (received from a queue by a consumer, but not yet deleted from the queue). If you reach this quota, Amazon SQS returns no error messages. If your queue has a large backlog of 20,000 or more messages with the same message group ID, FIFO queues might be unable to return the messages that have a different message group ID but were sent to the queue at a later time until you successfully consume the messages from the backlog." Link to the documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

jugg1es 6y ago
I only figured out what was happening by trial and error while trying to fix the production issue where I was moving 1000 messages at a time into a 2nd queue. Once I had dequeued 20k, everything starting pumping like normal. That's when I fired off an email to our AWS rep asking for confirmation from the SQS dev team. Within a couple weeks, the docs you just quoted were added.