6 ms·
I'm sure there remain good use cases for message buses that you have to run yourself, where there really are millions of messages that can't be batched up and n
by jonstewart 3y ago
I'm sure there remain good use cases for message buses that you have to run yourself, where there really are millions of messages that can't be batched up and need real-time and whatnot. But you can get pretty far with:
1. Write a bunch of records to an S3 object.
2. Trigger a lambda to process — infinite scale out!
and if a queue really is needed due to constrained consumers, then:
1. Write a bunch of records to an S3 object.
2. The lambda trigger puts a message into an SQS queue with the S3 object's URI.
3. An auto-scaling group gets the message off the queue and processes the data in the object.
If ordering's important, then the SQS queue can be made FIFO. That has pretty low numbers in messages/second, but since records are being batched into S3 objects you can still have fairly high throughput.
It used to be that the elegance of queues for such systems would tempt developers into the operational slough of despond that is running these types of systems. Again, I'm sure it's warranted for some applications, but S3, Lambda, and SQS as above work nicely together.
- pramodbiligiri 3y agoYou can get rid of Step 2 if you're going to be creating a new S3 object for each batch. AWS has an inbuilt feature to trigger SQS notifications for S3 operations: https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable...