7 ms·

AWS SNS vs. SQS – Main Differences

123 points by adrianomartins 4y ago

rbluethl 4y ago
Great writeup and comparison. Also, serverlessq looks really interesting. I've struggled with queues quite often in the past – can't wait to give this one a try.
- jack335 4y ago
  Thanks a lot! If you want, give ServerlessQ a try and let me know if you need any help! You find my contact info on my landing page: serverlessq.com or DM me on twitter: twitter.com/sandro_vol
philliphaydon 4y ago
SQS can push to lambda without polling.
- kondro 4y ago
  Technically Lambda polls SQS (minimum 5 pollers when used as an event source for Lambda). But you don't have to think too much about it. It's mostly important to remember when you're wondering why your mostly-empty queue is still costing you $0.26/month, even if you've got the receive message timeout set to 20 seconds.
  - jack335 4y ago
    Exactly. SQS is a queue where a message resides. Lambda basically polls this message. But like kondro said basically it doesn't matter (directly). I think it is important to understand the event source mapping takes over the polling. But in the end it looks like a push system. But the whole point about costs with empty queues is important. It is definitely important to understand if you want to customize queues for example with long polling. This parameter changes the time your lambda will poll from your queue
  - philliphaydon 4y ago
    Sure but it’s transparent. You don’t need to go write all that logic, against every queue, with possible bugs, handle different scenarios, etc. You just need to write the logic to process the message passed to the lambda. Under the hood it might be polling. But it gives the illusion it’s push and it’s bloody useful.
  - dkdbejwi383 4y ago
    If you use this though, you should be aware that the Lambda-SQS polling mechanism doesn't know about the concurrent execution limits for lambda. If you have set a limit on concurrent executions for your function, or are at the account-level limit, the Lambda-SQS polling mechanism will continue to poll for messages and invoke functions even if execution would fail. This will increment the receive count, and if you have set a limit on the receive count, you may find you drop messages or send them to the DLQ when your handler function hasn't yet had a chance to try and process a given message.
    - cloakandswagger 4y ago
      The Lambda-SQS poller will slow down in response to the lambda's error rate for this exact reason. Note that this can cause issues: say you have a time sensitive application that receives a batch of "bad" messages which cause failed lambda invocations. The poller will slow down and the throughput will drop drastically, even though your intention might be for the lambda to continue processing at the same rate and power through the bad messages. This behavior can be disabled with a support request.
- TameAntelope 4y ago
  You can also use Websockets and long polling, so it’s probably not accurate to say “SQS” is polling only.
traspler 4y ago
Pretty good and simple explanation, thanks a lot! There are a couple of things I‘m left wondering: When the SNS is used to notify through Email, Push or SMS then even in the fan-out pattern, no SQS is involved on these paths, right? (seems to me that only on A2A paths SQS would be involved, at least from the diagrams) So is there anything else to help with reliability there that the notifications actually do go out? For workloads where you e.g. want to alert users through different means, not every user might have the same options selected, e.g. some only Push and others SMS and EMail; would that be modeled through different topics which have different combinations of subscribers attached (some only 1) or would it be better to skip SNS and push multiple messages to different queues directly?
- jon-wood 4y ago
  An SNS message is delivered to a number of endpoints, which can include email, push notifications, SMS, or various AWS services. In its payload the content to send to each type of endpoint so you can make sure it’s going through in the right format.
- jack335 4y ago
  Hi author here :) Thanks for your kind words. Regarding fanout. Yes exactly. Fanout doesn't mean it needs to involve SQS just it can involve SQS. It is also called a fanout pattern if you do a A2P and only notify Emails, SMS, etc. To your second question how the architecture would look like for different preferences of architecture. I think the main benefit of that architecture is that customer can subscribe to a topic. That means if your user A subscribes to the topic for Email and not in-app notification that is fine. It would be also just the one topic. The consumer/subscriber has the power to subscribe and unsubscribe to topics (similar like you can to newsletters basically). That is one of the main benefits. With a queue the producer would need to define which consumer will get the message and most probably it will be another application. Does that help? :)
- zmgsabst 4y ago
  I think it may depend on your architecture, but naively: - SQS to send alert, all modes - Lambda reading that queue, filtering on user settings - SNS per mode of communication, eg email or text My thinking is that you’d want to filter on user preference early, to prevent repeated work. A benefit of this approach is you prevent combinatorial complexity if you have both selection on kinds of alerts delivered and way to deliver alerts: the Lambda can handle all of that based on the user settings. And still a single SNS per communication channel. Your gating Lambda can also implement other features, like volume aware decisions — where it eg, rejects “marketing” messages if there have been too many to a single customer recently while still allowing through “transaction” messages.
lysecret 4y ago
Good writeup. This is how I like my advertisement ;). Also pretty interesting, how he built the product: https://sandro.volpee.de/series/serverlessq-bip https://sandro.volpee.de/series/serverlessq-bip
- jack335 4y ago
  Thanks! Yes combining good content with advertising my product is the goal but I really focus on providing benefits with the content I create :) Since I build on top of these great services I want to show at least how I am utilizing everything.
SideburnsOfDoom 4y ago
In my experience, SQS queues attached to a SNS topic is a common configuration. Each subscribed queue has a worker or "worker group" that gets a copy of all messages. The topic has "zero or more" subscribed queues, and when sending to the SNS topic, you don't need to know how many subscribers there are right now. In many cases, it's more a flexible equivalent to write to a SNS topic; not directly to a SQS queue. The article calls this "the fanout pattern"
- jack335 4y ago
  Yes this is the fanout pattern - called at least in AWS context. It is pretty normal to use it like that to decouple it even more. Producer don't need to know to send it to which queues
- kcb 4y ago
  Yes this is also equivalent to the exchanges and queues in RabbitMQ. Exchange = SNS, Queue = SQS.
101011 4y ago
Nice writeup! It'd be helpful to mention that SNS allows you to setup a retry policy and also allows for you to configure a DLQ when retries are exhausted.
Nouser76 4y ago
>SQS has a many-to-one relationship. You can send messages to a queue from many different producers but only one consumer can be defined. A consumer is another application, most often some compute instances such as Lambda, EC2, or Fargate. My understanding is that you can have multiple consumers of an SQS through the use of visibility timeouts[0]. Once a message is consumed it is as if that message doesn't exist for all other consumers until it reaches a timeout period or is marked done by that consumer. You can also manually mark a message as being ready for other consumers. This moves the message back into the queue for the other consumers to see. I'm going to be linking this article to my team. We've been talking about moving to SNS/SQS/etc. and this article helps understand the use cases and distinctions better. [0]: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
- specialp 4y ago
  I would still view it as many to one. Visibility timeouts are for concurrency. Semantically speaking I would consider one consumer with n concurrent workers as one consumer function/service. In workflow terms, a SNS is a fanout, and SQS is a queue.
  - andrew_ 4y ago
    You can subscribe multiple lambdas to an SQS queue. It's not recommended, but it's doable. many-to-many or many-to-one depends on your choices of infrastructure.
    - elsurudo 4y ago
      Yes, but only one of them will receive the message.
      - andrew_ 4y ago
        Only one would receive a particular message, but all will receive messages.
andrew_ 4y ago
FWIW you can wire an SNS topic directly to a Lambda - But watch out for frequency rules around that setup. Too many posts to the SNS Topic and you'll run into rate limits and dropped messages. (This is also why most suggest going to a queue first)
- sakopov 4y ago
  You can publish to an SNS topic with an SQS queue subscribing to it and invoking your lambda. This provides durability and allows you to configure other nice things like dead-lettering for any message that fails in lambda. You can survive some pretty massive outages with a setup like this. Oh and also, as of recently you can re-drive messages in your dead-letter queue to a source queue via SQS console.
  - andrew_ 4y ago
    Fairly certain that ability has been around a few years. I left that comment there for folks who didn't know that was possible, as going the SNS -> SQS -> Lambda route is what is most popular and most written about.
    - ysuhaas 4y ago
      I think the "new" feature is doing it via the console, instead of manually redriving the messages. Used it recently, great improvement: https://aws.amazon.com/blogs/aws/enhanced-dlq-management-sqs/ https://aws.amazon.com/blogs/aws/enhanced-dlq-management-sqs...
kposehn 4y ago
I really liked this article because it explained the differences in a very understandable and relatable way, and also because it sold me on ServerlessQ. This is a great example of content marketing that uses a question to both answer and impart value in a very logical and compelling way. Through understanding the differences between SQS and SNS the reader will know how to use those in the future, get a clearer idea of the complexity, and be presented with a much simpler and still full-featured alternative at the end. Well done.
- jack335 4y ago
  Hi, author here. Thank you very much I really appreciate it! :)
jakswa 4y ago
If you're coming from a ruby/sidekiq/redis land, there is a sidekiq-inspired repo trying to make the transition easy https://github.com/ruby-shoryuken/shoryuken https://github.com/ruby-shoryuken/shoryuken
__david__ 4y ago
I'm confused about calling SQS a many to one service. One of the use cases I've seen (and that is endorsed by AWS) is a worker queue, where someone puts a job into the queue and there's a whole bunch of workers waiting. One of the workers grabs a message, processes it and then deletes it. That seems like a many to many service to me… Or maybe a many-to-one-of-many service?
- zyberzero 4y ago
  You're kind of on it. > One of workers the grabs a message, processes it and then deletes it. If I understood it correctly, in SQS, a message can only be handled by one worker, while SNS can have several workers processing the same message (let's say one worker ingesting the message for system A and another worker to ingest the same message in system B).

Great writeup and comparison. Also, serverlessq looks really interesting. I've struggled with queues quite often in the past – can't wait to give this one a try.

jack335 4y ago
Thanks a lot! If you want, give ServerlessQ a try and let me know if you need any help! You find my contact info on my landing page: serverlessq.com or DM me on twitter: twitter.com/sandro_vol

philliphaydon 4y ago

SQS can push to lambda without polling.

kondro 4y ago
Technically Lambda polls SQS (minimum 5 pollers when used as an event source for Lambda). But you don't have to think too much about it. It's mostly important to remember when you're wondering why your mostly-empty queue is still costing you $0.26/month, even if you've got the receive message timeout set to 20 seconds.
- jack335 4y ago
  Exactly. SQS is a queue where a message resides. Lambda basically polls this message. But like kondro said basically it doesn't matter (directly). I think it is important to understand the event source mapping takes over the polling. But in the end it looks like a push system. But the whole point about costs with empty queues is important. It is definitely important to understand if you want to customize queues for example with long polling. This parameter changes the time your lambda will poll from your queue
- philliphaydon 4y ago
  Sure but it’s transparent. You don’t need to go write all that logic, against every queue, with possible bugs, handle different scenarios, etc. You just need to write the logic to process the message passed to the lambda. Under the hood it might be polling. But it gives the illusion it’s push and it’s bloody useful.
- dkdbejwi383 4y ago
  If you use this though, you should be aware that the Lambda-SQS polling mechanism doesn't know about the concurrent execution limits for lambda. If you have set a limit on concurrent executions for your function, or are at the account-level limit, the Lambda-SQS polling mechanism will continue to poll for messages and invoke functions even if execution would fail. This will increment the receive count, and if you have set a limit on the receive count, you may find you drop messages or send them to the DLQ when your handler function hasn't yet had a chance to try and process a given message.
  - cloakandswagger 4y ago
    The Lambda-SQS poller will slow down in response to the lambda's error rate for this exact reason. Note that this can cause issues: say you have a time sensitive application that receives a batch of "bad" messages which cause failed lambda invocations. The poller will slow down and the throughput will drop drastically, even though your intention might be for the lambda to continue processing at the same rate and power through the bad messages. This behavior can be disabled with a support request.
TameAntelope 4y ago
You can also use Websockets and long polling, so it’s probably not accurate to say “SQS” is polling only.

traspler 4y ago

Pretty good and simple explanation, thanks a lot! There are a couple of things I‘m left wondering: When the SNS is used to notify through Email, Push or SMS then even in the fan-out pattern, no SQS is involved on these paths, right? (seems to me that only on A2A paths SQS would be involved, at least from the diagrams) So is there anything else to help with reliability there that the notifications actually do go out? For workloads where you e.g. want to alert users through different means, not every user might have the same options selected, e.g. some only Push and others SMS and EMail; would that be modeled through different topics which have different combinations of subscribers attached (some only 1) or would it be better to skip SNS and push multiple messages to different queues directly?

jon-wood 4y ago
An SNS message is delivered to a number of endpoints, which can include email, push notifications, SMS, or various AWS services. In its payload the content to send to each type of endpoint so you can make sure it’s going through in the right format.
jack335 4y ago
Hi author here :) Thanks for your kind words. Regarding fanout. Yes exactly. Fanout doesn't mean it needs to involve SQS just it can involve SQS. It is also called a fanout pattern if you do a A2P and only notify Emails, SMS, etc. To your second question how the architecture would look like for different preferences of architecture. I think the main benefit of that architecture is that customer can subscribe to a topic. That means if your user A subscribes to the topic for Email and not in-app notification that is fine. It would be also just the one topic. The consumer/subscriber has the power to subscribe and unsubscribe to topics (similar like you can to newsletters basically). That is one of the main benefits. With a queue the producer would need to define which consumer will get the message and most probably it will be another application. Does that help? :)
zmgsabst 4y ago
I think it may depend on your architecture, but naively: - SQS to send alert, all modes - Lambda reading that queue, filtering on user settings - SNS per mode of communication, eg email or text My thinking is that you’d want to filter on user preference early, to prevent repeated work. A benefit of this approach is you prevent combinatorial complexity if you have both selection on kinds of alerts delivered and way to deliver alerts: the Lambda can handle all of that based on the user settings. And still a single SNS per communication channel. Your gating Lambda can also implement other features, like volume aware decisions — where it eg, rejects “marketing” messages if there have been too many to a single customer recently while still allowing through “transaction” messages.

lysecret 4y ago

Good writeup. This is how I like my advertisement ;). Also pretty interesting, how he built the product: https://sandro.volpee.de/series/serverlessq-bip https://sandro.volpee.de/series/serverlessq-bip

jack335 4y ago
Thanks! Yes combining good content with advertising my product is the goal but I really focus on providing benefits with the content I create :) Since I build on top of these great services I want to show at least how I am utilizing everything.

SideburnsOfDoom 4y ago

In my experience, SQS queues attached to a SNS topic is a common configuration. Each subscribed queue has a worker or "worker group" that gets a copy of all messages. The topic has "zero or more" subscribed queues, and when sending to the SNS topic, you don't need to know how many subscribers there are right now. In many cases, it's more a flexible equivalent to write to a SNS topic; not directly to a SQS queue. The article calls this "the fanout pattern"

jack335 4y ago
Yes this is the fanout pattern - called at least in AWS context. It is pretty normal to use it like that to decouple it even more. Producer don't need to know to send it to which queues
kcb 4y ago
Yes this is also equivalent to the exchanges and queues in RabbitMQ. Exchange = SNS, Queue = SQS.

101011 4y ago

Nice writeup! It'd be helpful to mention that SNS allows you to setup a retry policy and also allows for you to configure a DLQ when retries are exhausted.

Nouser76 4y ago

>SQS has a many-to-one relationship. You can send messages to a queue from many different producers but only one consumer can be defined. A consumer is another application, most often some compute instances such as Lambda, EC2, or Fargate. My understanding is that you can have multiple consumers of an SQS through the use of visibility timeouts[0]. Once a message is consumed it is as if that message doesn't exist for all other consumers until it reaches a timeout period or is marked done by that consumer. You can also manually mark a message as being ready for other consumers. This moves the message back into the queue for the other consumers to see. I'm going to be linking this article to my team. We've been talking about moving to SNS/SQS/etc. and this article helps understand the use cases and distinctions better. [0]: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

specialp 4y ago
I would still view it as many to one. Visibility timeouts are for concurrency. Semantically speaking I would consider one consumer with n concurrent workers as one consumer function/service. In workflow terms, a SNS is a fanout, and SQS is a queue.
- andrew_ 4y ago
  You can subscribe multiple lambdas to an SQS queue. It's not recommended, but it's doable. many-to-many or many-to-one depends on your choices of infrastructure.
  - elsurudo 4y ago
    Yes, but only one of them will receive the message.
    - andrew_ 4y ago
      Only one would receive a particular message, but all will receive messages.

andrew_ 4y ago

FWIW you can wire an SNS topic directly to a Lambda - But watch out for frequency rules around that setup. Too many posts to the SNS Topic and you'll run into rate limits and dropped messages. (This is also why most suggest going to a queue first)

sakopov 4y ago
You can publish to an SNS topic with an SQS queue subscribing to it and invoking your lambda. This provides durability and allows you to configure other nice things like dead-lettering for any message that fails in lambda. You can survive some pretty massive outages with a setup like this. Oh and also, as of recently you can re-drive messages in your dead-letter queue to a source queue via SQS console.
- andrew_ 4y ago
  Fairly certain that ability has been around a few years. I left that comment there for folks who didn't know that was possible, as going the SNS -> SQS -> Lambda route is what is most popular and most written about.
  - ysuhaas 4y ago
    I think the "new" feature is doing it via the console, instead of manually redriving the messages. Used it recently, great improvement: https://aws.amazon.com/blogs/aws/enhanced-dlq-management-sqs/ https://aws.amazon.com/blogs/aws/enhanced-dlq-management-sqs...

kposehn 4y ago

I really liked this article because it explained the differences in a very understandable and relatable way, and also because it sold me on ServerlessQ. This is a great example of content marketing that uses a question to both answer and impart value in a very logical and compelling way. Through understanding the differences between SQS and SNS the reader will know how to use those in the future, get a clearer idea of the complexity, and be presented with a much simpler and still full-featured alternative at the end. Well done.

jack335 4y ago
Hi, author here. Thank you very much I really appreciate it! :)

jakswa 4y ago

If you're coming from a ruby/sidekiq/redis land, there is a sidekiq-inspired repo trying to make the transition easy https://github.com/ruby-shoryuken/shoryuken https://github.com/ruby-shoryuken/shoryuken

__david__ 4y ago

I'm confused about calling SQS a many to one service. One of the use cases I've seen (and that is endorsed by AWS) is a worker queue, where someone puts a job into the queue and there's a whole bunch of workers waiting. One of the workers grabs a message, processes it and then deletes it. That seems like a many to many service to me… Or maybe a many-to-one-of-many service?

zyberzero 4y ago
You're kind of on it. > One of workers the grabs a message, processes it and then deletes it. If I understood it correctly, in SQS, a message can only be handled by one worker, while SNS can have several workers processing the same message (let's say one worker ingesting the message for system A and another worker to ingest the same message in system B).