18 ms·
On SQS
- sessy 7y agoWE are heavy users of AWS. SQS is the only service where we have had zero downtime. The only downside we have about SQS is you can pull out only 10 messages at a time (without batching). You can have parallel readers but they result in some duplicates. There is SQS FIFO but it is throttled.
- tybit 7y agoPlus FIFO isn’t integrated with SNS nor is it integrated with Lambda leaving it in an island of its own within AWS.
- actuator 7y agoIt went down on us once for extended period in 2015. It was chaotic as you don't expect it to fail. If memory serves me right, even S3 suffered that day.
- Twirrim 7y agoA whole slew of AWS services went down that day. Tim's not wrong when he indicates that almost every service in AWS has a dependency on it (Amazon has services split up in to tiers based on how much they can rely on other services for critical components, SQS is pretty high up in the tiering.) I was on-call that day for an AWS service. There wasn't much I could do but sit muted on the conference call and watch some TV, waiting for the outage to be over.
- reallydude 7y agoNowhere is cost mentioned. Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. You can easily do what SQS does for so much cheaper (this includes horizontal scaling and failure planning), that I'm consistently surprised anyone uses it. Either you are running at a volume where you need high throughput and it's pricey, or you're at such a low throughput you could use any MQ (even redis). > Oh but what about ORDERED queues? The only way to get ordered application of writes is to perform them one after the other. This is another WTF. Talking about ordered queues is like talking about databases, because it's data that's structured. If you can feed data from concurrent sources of unordered data to a system where access can be ordered, you have access to a sorted data. You deal with out-of-order data either in the insertions or a window in the processing or in the consumers. "Write in order" is not a requirement, but an option. Talking about technical subjects on twitter always results in some mind-numbingly idiotic statements for the sake of 144 characters.
- obituary_latte 7y agoIn the first subheading, towards the bottom, the author mentions it’s $0.40 per million api calls.
- cldellow 7y agoS3's eventual consistency aside, how is it cheaper? It would seem to me that naively, S3 charges $5 per million POST requests, so it's 10x worse than SQS's $0.40 per million.
- reallydude 7y agoWriting directly from EC2 is free (within same region).
- etaioinshrdlu 7y agoOnly the data transfer is free. The API requests are not free. S3 does come out to be more expensive...
- reallydude 7y agoOnly if you are pushing 1 message per write. 1 write per agent per minute, ends up at just over 2$ per month fox 10 agents, regardless of the number of messages - it's cheaper for any setup I've encountered. AWS services are always middling solutions, which you can often optimize for better cost efficiency.
- etaioinshrdlu 7y agoInteresting point. Should have mentioned this difference though, it's a quite different architecture with different tradeoffs.
- kondro 7y agoThis is a message queue. But wouldn’t be fit for purpose for the way most people need to use message queues.
- etaioinshrdlu 7y agoI really wish SQS had reliably lower latency, like Redis, and also supported priority levels. (Also like redis, now, with sorted sets and the https://redis.io/commands/bzpopmax https://redis.io/commands/bzpopmax command.) Has anyone measured the performance of Redis on large sorted sets, say millions of items? Hoping that it's still in single digit milliseconds at that size... And can sustain say 1000QPS...
- plasma 7y agoWe use Redis as a job queue and its great; the only limitation is being sometimes concerned about job queue size due to memory limits of the Redis server itself.
- espadrine 7y agoAlso, when you don't want to lose a message, the Redis persistence story requires careful thought. It requires setting up RDB + AOF + appendfsync=always + backups.
- actuator 7y agoThis would just kill the performance on virtualized hardware like EBS. You would lose a lot of benefits of Redis at that point.
- manigandham 7y agoYou don't need to fsync on every write, that's what the replica is for. At lower/mid-tier hardware, network is faster than storage and your message is on multiple machines before it's even written to disk so fsync of 1 second is usually fine.
- ramraj07 7y agoAlways hear using redis etc for job queues, what do these "job queues" entail? I've been an amateur and have used postgres tables as queues .. am I not being efficient?
- varelaz 7y agoOne important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.
- philliphaydon 7y ago> you can read the same message twice from different workers If you have a short visibility timeout that it appears back on the queue before you delete it... Sure.
- varelaz 7y agoNo. By default SQS uses `at-least-once delivery` system. It's distributed and when you connect to API one SQS node that you working with could have different state comparing to another, so you may fetch the same message twice by 2 separate workers even with big visibility timeout.
- lclarkmichalek 7y agoMaking the processing of a message idempotent is the ideal way to handle that limitation
- cies 7y agoThat's not always possible. Thus a big no-no for SQS if that's what you need. Concluding: SQS cannot replace RabbitMQ in all usecases.
- tybit 7y agoIf you’re message consumer isn’t idempotent then no MQ can help you. Exactly once delivery is impossible other than with at least once delivery and an idempotent consumer. https://bravenewgeek.com/tag/amazon-sqs/ https://bravenewgeek.com/tag/amazon-sqs/
- andrewstuart 7y agoI use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.
- varelaz 7y agoThere are VPC API endpoints for this. As far as I recall it was all the time there from VPC begging, but I might be wrong.
- andrewstuart 7y agoOnly since December 2018 and you need to pay extra... https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-sqs-vpc-endpoints-aws-privatelink/ https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-sq...
- LunaSea 7y agoDo you have an example of such a queue somewhere? I think I have a rough idea about how it works because I implemented something similar about four years ago in PostgreSQL but kept getting locking issues I couldn't get out of: - https://stackoverflow.com/questions/33467813/concurrent-crawler-postgresql-query-issues https://stackoverflow.com/questions/33467813/concurrent-craw... - https://stackoverflow.com/questions/29807033/postgresql-concurrent-transaction-issues https://stackoverflow.com/questions/29807033/postgresql-conc... Also, what kind of queue size / concurrency on the polling side are you able to sustain for your current hardware?
- sorokod 7y agoHere https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/ https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock... is a good discussion.
- andrewstuart 7y ago
- cyberferret 7y agoWe've used SQS with great results (and reliability) for many years now, but I am interested to hear the author talking about 'replaying queues' to replicate faults. I never realised you could do this with SQS. Or can you? I thought once a queue item was processed and deleted, that was it, it was gone forever, but perhaps you can see historical queue data somewhere? (without having to store it yourself)
- ilogik 7y agomaybe once you've processed a message, you resend it to a different queue, just in case?
- cyberferret 7y agoThat's actually a neat idea. Since we don't have to pay for number of items in a queue, I am thinking we could create a dummy queue and just handball all incoming messages to it for safekeeping (at least for 14 days)?!?
- kondro 7y agoOr send everything to SNS and have two SQS queues subscribed to the topic.
- hexene 7y agoPossibly this: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
- cyberferret 7y agoWe actually already use the DeadLetterQueues in our service at the moment, but these are when failed SQS deliveries happen, then they get replicated to the DLQ. I am more interested in diagnosing successful SQS deliveries after the fact, to see what the payloads were in case there was a downstream problem. It seems that SQS deliveries that don't get a 200 response from our service go to the DLQ, but those that get a successful 200 disappear into the ether.
- redact207 7y agoAuthor of https://node-ts.github.io/bus/ https://node-ts.github.io/bus/ here. SQS is definitely one of my most favourite message queues. The ability to have a HA managed solution without having to worry about persistence, scaling or connections is huge. Most of the complaints are native to message based systems in general. At least once message receives, out of order receives, pretty standard faire that can be handled by applying well established patterns. My only request would be to please increase the limits of message visibility timeouts! Often I want to delay send a message for receipt in 30 days. SQS forces me to cook some weird delete and resend recipe, or make this a responsibility of a data store. It's be really nice to do away with batch/Cron jobs and deal more with delayed queue events.
- plasma 7y agoRE: visibility timeout beyond 30 days, you may be more after a “saga” that has state and is long running (hours/days/months/years). You can imagine building a saga system on top of a queue system.
- redact207 7y agoYou're absolutely right, in fact I have a whole package that is just that https://node-ts.github.io/bus/packages/bus-workflow/ https://node-ts.github.io/bus/packages/bus-workflow/. The problem is this. Let's say that I want to trigger a step in a "free trial" saga that sends an email to the customer 10 days after they sign up nudging them to get a paid account. If I can delay send this message for 10 days then it's easy. However because SQS has a much shorter visibility timeout, I have to find a much more roundabout way of triggering that action.
- plasma 7y agoYeah, that makes total sense. For some of our saga's (we don't use SQS -- we use a custom redis queue), we have the saga potentially wake up and immediately sleep again ("Nothing to do right now, defer again in a few days"). But yes, a quirk.
- 7y ago
- plasma 7y agoAnyone run a multi-tenant SaaS and handle fairness with jobs “fairly”? Occasionally we use to have all workers tied up on a single customers long running tasks, we mitigated by using a throttler we wrote that can defer a job if too many resources are in use by the customer, but it’s not ideal. I’d love a priority based, customer throttled (eg max concurrent tasks) queue. We can prioritize by low/medium/high using separate queues, and could make a set of queues per customer; but that is starting to explode how many queues we have and feels unmanageable.
- varelaz 7y agoWe implemented this with additional DB checks. For example: we put only one job to queue at a time per customer, the rest are in database until one that in progress is not processed. Actually most of the prioritization could be implemented through additional DB. With SQS in most cases you need persistent reflection of job to keep its status, process times, results. You can put to queue only few items that are highest priority to guarantee that workers are busy next 10-30 minutes.
- plasma 7y agoThanks for the suggestion, ill reflect on that; we were also considering an "overflow" queue that would receive jobs if there was a recent high insert rate of a customer's jobs to the main queue, but that didn't solve the "cost" problem of a job potentially being large. Was hoping to avoid having "side car" infrastructure for this, but I don't think I can escape it :)
- rtpg 7y agoUsing a database lets you make much better decisions on this space Tbh purpose-built queues are taken way too eagerly by programmers who later end up needing the flexibility offered by a more general data store.
- cookiecaper 7y agoYes. I've seen it in all kinds of teams. Anything that allows a developer to retrieve some data after a local server restart essentially gets treated by devs as a system of record, regardless of the intricacies or guarantees involved. My personal experience is that abuse of queuing/messaging systems along this axis is rampant. Engineering leaders must keep a close eye on how these types of mechanisms are utilized to ensure things don't go off the rails. I've seen far too many serious data loss events that boil down to "we lost our AMQP queue". It's critical that developers understand the limitations of the systems that run their code rather than just jumping aboard that "SQL is for old people" hype train.
- kayoone 7y ago> Those messages will age out and vanish after a little while (14 days is currently the max); but before they go, they’re stored carefully and are very unlikely to go missing can somebody expand on this? I know about the 14 days limitation but this makes it sound like you can store messages for a long time and still recover them somehow?
- actuator 7y agoI think what he meant was, Once you successfully put a message to SQS, it is kept till the 'message retention period' unless you explicitly call delete on it. Right now you can configure the period to be upto 14 days. AFAIK, there is no way to recover messages older than their retention period.
- kayoone 7y agoYou are right, thank you! I misinterpreted that.
- TheAceOfHearts 7y agoThey're saying you've very unlikely to experience data loss after a message has been saved by SQS. Presumably they have multiple layers of redundancy at varying levels in order to provide this guarantee.
- hexene 7y agoOne downside of SQS is that it doesn't support fan-out, for eg. S3->SQS->multiple consumers. The recommendation instead seems to be to first push to SNS, and then hookup SQS/other consumers to it. Kinesis/Kafka would appear to be better suited for this (since they support fan-out like SNS and are pull-based like SQS), but aren't as well supported as SNS/SQS (you can't push S3 events directly to Kinesis for eg.) Can someone from AWS comment on why that is? Also, related: when can we expect GA for Kafka (MSK)?
- actuator 7y agoGenerally they do GA in the next ReInvent from when as service is announced, so probably by end of year. But I won't be sure on MSK. It is extremely limited right now, last time I checked their API had no way for even changing the number of nodes.
- staticassertion 7y agoI do S3 -> SNS -> SQS. I don't see why I would use Kinesis instead. The SNS bit is totally invisible to the consumers (you can even tell SNS not to wrap the inner message with the SNS boilerplate), downstream consumers just know they have to listen to a queue. I don't see a downside to this approach. Perhaps some increased latency?
- hexene 7y agoIf you wanted multiple pull-based consumers for the stream, wouldn't you need a separate SQS queue per consumer, with each queue hooked up to SNS? Perhaps I'm mistaken, but that seems brittle to me. With Kinesis/Kafka, you only need to register a new appName/consumer group on the single queue for fan-out. Plus, both are FIFO by default, at least within a partition.
- staticassertion 7y agoThat's exactly how you do it. To me, it's the opposite of brittle - every consumer owns a queue, and is isolated from all other consumers. Clients are totally unaware of other systems, and there's no shared resource under contention.
- edoo 7y agoI recreated an SQS style service on AWS for sole purpose of avoiding being locked into Amazon. It used their autoscale system and was about the dumbest simplest api you can imagine, just depositing messages into a geographically replicated db table for later processing. We also used it as sort of a backup system, where the messages were never truly deleted and everything could be reprocessed as needed since the front end db that held processed records was an absolute amateur disaster just waiting for some malicious sql injection wipes. It was solid as a rock and extremely low maintenance. I think things have become relatively standardized since then (aka lots of duplicate SQS compatible api/services), so unless there was a serious requirement to stay off SQS I'm not sure I'd do it again.
- gregw2 7y agoWith 4-5 concurrent processes reading from my (FIFO?) SQS queue, I have a hard time seeing the contents of the queue from the console. SQS really doesn't handle the concurrent readers well in my experience.
- pojzon 7y agoAfter reading some comments in this thread im concerned about how many misconceptions people have about AWS services. Most of the stuff from "correcting comments" is plain available for anyone.
- oceanbreeze83 7y agois SNS + SQS a reasonable solution for realtime irc style topic chatrooms?
- Cthulhu_ 7y agoIt depends, but you can safely assume neither service is intended for real-time use cases. They'll be fast, but there won't be any speed / delivery guarantees.
- cyberferret 7y agoThere is also a size limit on the messages sent via SQS - 256Kb IIRC. Might be OK for small text based messaging, but if you want to talk attachments and other payloads then you have to hook it up to other services like S3 etc. and then it becomes complicated. Also, as another respondent replied - there is no real deliverability guarantee, although there are certain ways to handle that within SQS.
- bpicolo 7y agoMessage fanout isn’t really solved there. Also probably quite expensive for the task
- appwiz 7y agoAWS AppSync (https://aws.amazon.com/appsync/ https://aws.amazon.com/appsync/) is a better fit for the chat room use case because of server pushed events over a persistent connection (WebSockets). Launch the Chat sample in the console to try it out.
- Kiro 7y ago> So what to do? In almost all cases it's better to propagate failure and backpressure. This gives a chance for the upstream to decide if it wants to retry or fail itself. It keeps things simple and easy-to-understand. It doesn't build up huge backlogs that become disasters. What does this actually mean in practical terms?
- owenmarshall 7y agohttps://ferd.ca/queues-don-t-fix-overload.html https://ferd.ca/queues-don-t-fix-overload.html explains it better than pretty much anything I’ve read on the topic.
- ejcx 7y agoWay to go Rick! You know you've made it when people write blog posts about your hot takes on Twitter
- yilugurlu 7y agoHas anyone ever measured the latency of the sending message to SQS? I was using with ELB in t2.medium instances, and my API (handle => send message to queue => return {status: true}) response times were around 150 - 300 ms and replaced SQS with RabbitMQ, and it went down to around 75-100 ms. Does anyone think that sending message to SQS is slow? Edit: With this update, I was able to process almost 3 x requests with the same resources, and it lowered my bills quite a lot. For example my SQS bill for last month Amazon Simple Queue Service EUC1-Requests-Tier1 $0.40 per 1,000,000 Amazon SQS Requests per month thereafter 290,659,096 Requests $116.26 it went to 0, and ec2 cost went down as well because ELB spun up fewer instances that I could handle more quest with the same resources. This was my experience with SQS. I just wanted to share it.
- avh02 7y agoBenchmarks online show this to be true, depends on your use case if it's acceptable I guess.
- Feeble 7y agoWere you running RabbitMQ clustered with persistent queues? I don't think SQS is primarily for low-latency messaging, but rather a provided high available MQ with very little hassle.
- yilugurlu 7y agoI wasn't, single instance in the same subnet with persistent work queues.
- KirinDave 7y agoIn cases where latency is of primary concern, don't use SQS. Consider also not using Rabbit as its tail latency isn't hugely more reliable when loaded.
- actuator 7y agoUmm.. is this in the same DC? From what I remember from numbers from New Relic. - The average SQS Put latency is: 10-20 ms. - RabbitMQ Publish Message in the same DC should not be more than the round trip time of a DC assuming a persistent connection. So, about 0.5 ms
- polskibus 7y agoDoes anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex than customers deem acceptable. I know some people use Akka with Persistence module, but I would welcome other alternatives.
- notduncansmith 7y agoRedis fits this niche for me most of the time, but I always start in-process for simplicity. If durability is all you need, SQLite is a great reliable option.
- wmfiv 7y agoDepending on the project a database (Postgres) can make a nice database, queue and cache. You get transactions among all three which which makes life much simpler in the beginning and it will happily march along processing thousands or tens of thousands of TPS. You can scale by adding read replicas and eventually moving to separate databases for each purpose. If you're on the JVM then ActiveMQ (and other Java queues) will usually run embedded and have options to use a database for persistence.
- gizzlon 7y agoGuess it depends on the definition of "queue". Potentials: - https://nsq.io/ - https://nats.io/
- dantillberg 7y agoI've worked with SQS at volumes in thousands of messages per second with varied (non-tiny) payload sizes. SQS is a very simple service, which makes it fairly reliable, though part of the reason for the reliability is that the API's guarantees are weak. And it can be economical, but I've had to build a lot of non-trivial logic in order to interact with SQS robustly, performantly, and efficiently, especially around using the {Send,Receive,Delete}MessageBatch operations to reduce costs. With the caveat that I think my use case has been quite different from what's discussed in this article, here are some of the problems I've encountered: - Message sizes are limited, but in a convoluted way: SendMessageBatch has a 256KiB limit on the request size. Message values have a limited character set allowed, so you need to base64-encode any binary data. This also means that there's not exactly a max message size; you can batch up to 10 messages per SendMessageBatch but not in excess of 256KiB for the whole request. - If you want to send more than 256KiBx3/4-(some padding) or around 180KiB of data for any single message, you need to put that data somewhere else and pass a pointer to it in the actual SQS message. - SQS does routinely have temporary (edit: partial) failures that generally last for a few hours at a time. ReceiveMessageBatch may return no messages (or less than the max of 10) even if the queue has millions of messages waiting to be delivered; SQS knows it has them somewhere but it can't find them when you ask. And DeleteMessageBatch may fail for some of the messages passed while succeeding for others; it will sometimes fail repeatedly to delete those messages for an extended period. - The SDKs provided by AWS (for either Java or Go) don't help you handle any of these things well; they just provide a window into the SQS API, and leave it to the user to figure all the details out.
- timbray 7y agoUm, I’ve been at AWS since late 2014, and AFAIK the only extended SQS hiccup correlated with the DynamoDB issue in 2016. SQS isn’t perfect but I’m pretty sure “does routinely have temporary failures that generally last for a few ours at a time” is just wrong.
- noelherrick 7y agoI believe GP was talking about particular messages failing, not a total system outage. In my use of AWS, the status page almost never reports an outage even though that AWS service is down for me-as in the most I've ever seen is some hand wavey message that there's elevated error rates. So you could be right, SQS hasn't failed entirely, but that probably means there's a good number of failed requests that are below the margin where AWS would consider it down.
- KirinDave 7y agoI see lots of engineers who argue that queues are bad and cause more problems than they solve. "Queues don't have backpressure" is a common complaint. It wouldn't be a difficult thing to add to a queue interface, but people don't do it despite the relative ease of the task. It is pretty easy to get a cardinality estimate on the queue size of any given queue even as it's rapidly increasing and decreasing in value. The reason why is that it's probably a bad idea in the first place to mix your control and data interfaces into an inextricable knot. Since you're not going to be able to institute rate limiting at a sub-millisecond latency in a distributed system anyways, why isn't your control plane separate and instrumented? Once you have a separate control plane, you can introduce back-pressure in many different ways and do so with a better understanding of the dissemination of those throttling values throughout your system. So what I see are engineers who are actually ignoring the the architectural decision with at least as many implications as "queue vs diffuse interface," namely "control and data or just one monolithic system."
- mabbo 7y agoA long time ago, as new-ish developer, I was building a system that needed to take inputs, then run "pass/fail/wait and try again later" until timeout or completion. This wasn't mission-critical stuff, mind you, so a lost message would annoy someone but not cause any actual harm. As I was figuring out how to setup a datastore, query it for running workflows and all that jazz, I happened upon an interesting SQS feature: Post with Delay. And so, the system has no database. Instead, when new work arrives it posts the details of the work to be done to SQS. All hosts in the fleet are polling SQS for messages. When they receive one, they do the checks and if the process isn't complete they repost the message again with a 5-minute delay. In 5 minutes, a host in the fleet will receive the message and try again. The process continues as long as it needs to. Looking back, part of me now is horrified at this design. But: that system now has thousands of users and continues to scale really well. Data loss is very rare. Costs are low. No datastore to manage. SQS is just really darned neat because it can do things like that.
- bkanber 7y agoWhy are you horrified at a design that works well, scales well, is resilient enough for its use case, and is low cost? The whole point of an engineering design process is to find designs that meet these types of requirements. Honestly, this sounds like the perfect solution for what you're trying to accomplish.
- mabbo 7y agoBecause looking at it now, something feels deeply wrong about it, haha. Honestly, if I'd used a database it probably would have opened up a few more options for future work. I can't do any analytics about how long things typically take, who my biggest users are, etc. I mean, I could, but I'd have to add a datastore for that. Adding new details to the parameters of the system requires very careful work to make all changes backwards and forwards compatible so that mid-deployment we don't have messages being pushed that old hosts can't process or new hosts seeing old messages they don't understand. That's good practice generally, but it's super mission critical to get right this way. Also, a dropped message is invisible. SQS has redrive, sure, and that helps but if there were a bug, an edge case, where the system stopped processing something and quietly failed, that processing would just stop and we'd never know. If the entries were in a datastore, we'd see "Hey, this one didn't finish and I havne't worked on it lately, what gives?".
- Jemaclus 7y agoWe love SQS, but one of the problems we're running into lately is the 256kb per message limitation. We do tens of millions of messages per day, with a small percentage of those reaching the 256kb limit. We're approaching the point where most of our messages will hit that limit. What are our options for keeping SQS but somehow sending large payloads? Only thing I can think of is throwing them into another datastore and using the SQS just as a pointer to, say, a key in a Redis instance. (Kafka is probably off the table for this, but I could be convinced. I'd like to hear other solutions first, though.
- somedev55 7y agoWe point to an s3 object for any large payloads
- m4r 7y agoHard to suggest smth without details, but imo 256kb per message is a lot and enough. One thing I would consider is storing such "messages" payload on s3, and sending notification to sqs allowing you to fetch given file and process it (this however might be expensive if you happen to have such high traffic)
- mark242 7y agoThe Java SDK for SQS automatically drops >256k messages into an S3 bucket and stores a pointer to the message in SQS itself. You set the bucket and the client transparently handles retrieving a message from SQS/S3 when necessary. Based on the structure of the message (UUIDv4) you could probably roll your own implementation in any language.
- staticassertion 7y agoI think this is a decent response - they really nail what @rbranson misses, that the failures he mentions are actually features we're after. An example, > Convert something to an async operation and your system will always return a success response. But there's no guarantee that the request will actually ever be processed successfully. Great! I don't want service A to be coupled to service B's ability to work. I want A to send off a message and leave it to B to succeed or fail. This separation of state (service A and B can't even talk to each other directly) is part of what makes queues so powerful - it's also the foundation of the actor model, which is known for its powerful resiliency and scalability properties. The author's suggestion of using synchronous communication with backpressure and sync failures is my last ditch approach. I have to set up circuit breakers just to make something like this anything less than a total disaster with full system failure due to a single service outage. Like the author, the "good use cases for queues" is very nearly 100% for me. I believe you should reach for queues first, and it's worth remodeling a system to be queue based if you can help it. Sometimes modeling as synchronous control is easiest, but I'm happy that I can avoid that in almost every case.
- dmix 7y ago> Convert something to an async operation and your system will always return a success response. It's funny reading this after using Erlang/Elixir over the last few years. The default is always async with the assumption it will fail - as async processes failing is a core part of the OTP application architecture. It's not something to be feared but a key part of how your application data-flow works.
- haolez 7y agoI've been planning on giving Erlang/Elixir a try, but we are very reliant on serverless and managed cloud services (like SQS) and I get the impression that managing a cluster of worker nodes for Erlang/Elixir would be too much work for us, since we would have to manage the servers, security patches, plan its scaling, etc. Maybe I'm wrong and it's not so much work in the end. Hoping for some feedback.
- curryst 7y agoFrom the operations side, I feel like SQS is a very sharp pair of scissors. In the hands of a good tailor, they'll make amazing things. On the other hand, most people use them to cut things they probably shouldn't, or just flat out Sprint around with them on a wet pool deck with their shoes untied. A non-comprehensive list of ways I've seen my developers shoot themselves in the foot: * Giant try-catch block around the message handling code to requeue messages that threw an exception. They neglected to add any accounting, so some messages would just never process. No one noticed until they saw the queue size never dropped below a certain amount during debugging. * Queue behavior is highly dependant on configuration. Bad queue configurations result in dropped messages. Queueing systems provide few features to detect and alert on these failures (it's rally not their job), but building a system to track the integrity of the business process across queues is deemed to onerous. * The built-in observability is generally not enough to be complete. I haven't seen a lot of great instrumentation libraries for SQS like there are for HTTP, meaning that observability is pushed on to the developer. They typically ignore that requirement because PMs rarely care until they realize we're unable to respond to incidents effectively. * Most people vastly overestimate their scale. The number of applications I've seen built on SQS "because scale" that end up taking less than 100 QPS globally is significant. Anecdotally, I would say the majority of queue-based apps I have seen could have solved their scaling issues within HTTP. * Many people want to treat queued messages like time-delayed HTTP requests. They are not, the semantics and design are totally different. I have seen people marshal requests to Protobuf, use it as the body of a message, and had another service read and process the request, and write another message to a queue that the first app reads back. It's basically gRPC over queues. Except that it solves none of the problems gRPC does, and creates a lot of problems. Just an example, how do you canary when you can't guarantee that the version of the app that sends the request will get the response to that request? I think SQS is an amazing tool in the hands of people that know when to use it, and how to use it. But my experience has been that most people don't, and the ecosystem to make it available to people who aren't experts just doesn't exist yet.
- cle 7y agoI agree with most of this. If you have non-trivial message handling logic in a production system, you probably shouldn't use SQS directly to drive your work. Your SQS handling logic should be simple and reliable. In most cases, if the handling logic is complex, long-running, or needs operational visibility (logging, monitoring, etc.), I'd write the message handler itself to just kick off a workflow via Step Functions or some other workflow system. You'll pay for that in initial development costs, because it is more complicated (you need to write lambda handlers, wire them up with CloudFormation, etc.), but the tradeoff is that it gives you a central place to look at each unit of work, instead of having your artifacts scattered around various logs (if at all). The takeaway for me is: distributed systems are hard. If you have distributed workers, you have entered into a vastly more complex realm. SQS gives you some tools to work successfully in that environment, but it doesn't (and can't) get rid of that complexity. Most of the problems I've seen relate to engineers not understanding the fundamental complexity of coordinating distributed work. Your choice of tech stack for your queues isn't going to make a big difference if you don't understand what you're fundamentally dealing with.
- danielmg 7y agoI clicked the link. Laptop set to loud by mistake. The cat asleep on the keyboard nearly died.
- daffjeri 7y agoStart earning money with paid surveys, etc. All this is free and easy. http://www.clixsense.com/?8172285 http://www.clixsense.com/?8172285