6 ms·
Amazon SQS is 15 Years Old
- deleted 5y ago[deleted]
- stephenhuey 5y agoEven though I first created images for EC2 in 2008, I've still never used SQS. I have a lot of experience with Google PubSub and this blog post is making me wonder if I should try leveraging SQS in production. I'm sure it's very mature, but I'm curious about how it feels from a developer perspective with a heavy production load.
- yzmtf2008 5y agoIMO, SQS is one of the best products that AWS offers. Airbnb built a queueing + scheduling system on SQS to great success [0] (disclaimer, I wrote this post). There are many others, e.g., Slack, that are building on top of SQS. [0]: https://medium.com/airbnb-engineering/dynein-building-a-distributed-delayed-job-queueing-system-93ab10f05f99 https://medium.com/airbnb-engineering/dynein-building-a-dist...
- lloydatkinson 5y agoYou mentioned the tick rate of the scheduler? What is it? Every second, every minute, etc?
- Supermancho 5y ago> SQS is one of the best products that AWS offers. I would never trust anyone who says this, regardless of who did what with it. SQS Standard (which was the only for a long time) has historically been a nightmare for almost any use case including job queueing. I love creating a whole separate cache to ensure when SQS delivers message multiple times, that I don't act on them. That's just ONE issue that I don't need to have. When you get into FIFO to avoid that, now your costing is ridiculous at scale AND the (relatively small) number of issues with FIFO that are still outstanding . eg https://tomgregory.com/3-surprising-facts-about-aws-sqs-fifo-queues/ https://tomgregory.com/3-surprising-facts-about-aws-sqs-fifo...
- arduinomancer 5y agoI don't really understand this point. I've found its trivial to just have a unique ID in the message and just check if the ID already exists in a jobs table to avoid double processing. I pretty much always have some metadata table needed for the job anyway, so its not like I'm building something extra.
- gunapologist99 5y agojobs table? why not just use the database directly then?
- chris_j 5y agoUse the database directly? I hope you're not suggesting sharing a database between services...
- gunapologist99 5y agoSharing an SQS queue between services is ok then? Even the most basic example for SQS queues from the AWS SA training course recommends using multiple queues for different classes of video transcoding. All of those benefits evaporate once you share a single queue; if you're just reading that queue to create a job in a database, then the queue provides no benefit.
- chris_j 5y agoI'm not quite clear what you mean by "sharing a single queue between services". If you mean having multiple different services reading from the same queue then no, I wouldn't do that. If you mean having a sending service and a receiving service share the same queue (either directly or indirectly) in the sense that the sender sends a message and the receiver reads it then I can't see any alternative to doing that. If I've misunderstood then would you be able to share what you were thinking of? I interpret arduinomancer's post above to mean that if the receiving service must action a single message once and only once (which is sometimes but not always necessary) then you need to give each SQS message a unique ID and that ID needs to be stored in as DB by the receiving service. I've used that pattern a few times and it works well (and doesn't result in unnecessary coupling between sender and receiver). In the past, I've used messaging systems that gave stronger transactional guarantees, but those systems were things like JMS or MQSeries and I don't really want to go back to those days.
- deleted 5y ago[deleted]
- kchoudhu 5y agoMost consistently excellent AWS product: it's a simple thing that does a simple job with minimal hassle. I suspect it will save me the trouble of ever having to run messaging infrastructure again.
- paulddraper 5y agoIt's got some pretty weird quirks. Like you can only send certain character sets. I had a very angry customer, and in the end it was some character (0x07?) that couldn't be sent via SQS. I had to either be certain that contents would never contain an SQS-unallowed character, or base64-encode all my contents.
- ElViajero 5y agoSimple Queue Service (SQS) It's just polite to write the meaning of acronyms.
- yummybear 5y agoAs a side note - For someone not totally an expert in AWS it’s sometimes difficult to choose between SQS, SNS, EventBridge and the list of other related technologies.
- ransom1538 5y agoSQS - pull SNS - push Sure you can mix and match and get all confused. But I would stick to that.
- deleted 5y ago[deleted]
- orf 5y agoNot really, more like SQS - 1 to 1, SNS - 1 to many
- commandlinefan 5y agoWe had to do some queuing for replication and priced SQS vs. Kinesis and Kinesis was orders of magnitude cheaper and did pretty much the same thing.
- paulddraper 5y agoSQS and MQ are pull and 1:1. SNS and EventBridge is push and 1:many. Kinesis and MSK is pull and 1:many.
- tforcram 5y agoI interned on the SQS team back in 2006 and subsequently joined the team full time for a few years including briefly managing it. I have some great memories of that time. Keeping things 'simple' while scaling as AWS grew was a great introduction to the industry.
- jtchang 5y agoThe last time I used SQS it was kinda expensive because I had to poll every minute or so it ate up the free tier. If I wanted to poll faster (like every 5 seconds) it was even more expensive. Maybe not the best use for my use case.
- lloydatkinson 5y agoWhy poll it? Doesn't your client library of choice support callbacks/events/observables etc?