5 ms·

I guess it's up for debate. While I agree some are silly (Elastic Beanstalk/Kinesis), renaming the product does not necessarily increase discoverability, which

I guess it's up for debate. While I agree some are silly (Elastic Beanstalk/Kinesis), renaming the product does not necessarily increase discoverability, which is a bigger issue. As for this list, I find a lot of these too dumbed down or missing some huge use-cases. S3 is simply an object store, which is a fairly common term. So "Unlimited FTP Server" is less obvious to me, and sounds old, outdated, and a security risk (although S3 can be that, too). Lambda is functions-as-a-service, or serverless. "App Scripts" would also tell me nothing, or sound like something for Office. SNS main use-case for us is not to "send mobile notifications, emails and/or SMS messages", but to link two systems together where some loss is acceptable, so "Amazon Messenger" is worse. SQS and SNS fit together well.

nostrebored 8y ago
I think the sqs and sns naming convention is a bit confusing. Realizing that sns should be the typical choice for an event based architecture is weird to people used to working with a message queue
- scarface74 8y ago
  SNS and SQS works together but serve two different purposes. SNS is for producers. You use SNS to say something happened. You have no guarantee that any consumer actually successfully processed the message and if the consumer is down, that message is lost to them. The only way a consumer can actually directly process a message is by subscribing to via http or lambda. But again, if the consumer errors or is down, you’re out of luck. SQS is a traditional simple queueing mechanism. It has no fanout capability on its own but you do get the traditional queuing functionality. But it doesn’t make sense logically for more than one process (or group of processes doing the same thing) to consume the queue. If you want the traditional fanout, filtering, multiple queues that do different things on the same event/message, you use SNS and SQS together.
  - cthalupa 8y ago
    >If you want the traditional fanout, filtering, multiple queues that do different things on the same event/message, you use SNS and SQS together. For this sort of use case, I've generally opted to go with a Kinesis stream, and I'm having a bit of a hard time understanding why a mix of SNS and SQS would be better here.
    - scarface74 8y ago
      For a simple “something happened” with a message and attributes, Kinesis is overkill and not as flexible. You can send an SNS message with attributes and subscribe to that SNS message with any combination of SQS queues, lambda functions, emails, http endpoints, etc and with any of the subscriptions you can design it so that any of the subscriptions only get messages based on attribute conditions. Also with SQS, you get the standard granular retries, dead letter queues etc. Yes with Kinesis you can do shards but it really doesn’t make sense to have more than one process reading messages from the same shard to scale out processing. With SQS, you can autoscale instances to read from the queue based on the queue size or just subscribe the SNS to an SQS queue and then subscribe the SQS queue to a lambda and let AWS work it’s magic.
      - cthalupa 8y ago
        Much appreciated! This actually has me rethinking the architecture on a project I'm working on right now. It looks like Kinesis would be a little bit cheaper at the volume of data I'm looking at... But the SNS/SQS method will let me sidestep some potential future scale-up concerns I had with the 5 reads per second limit without making a Rube Goldberg machine of Kinesis Analytics feeding into additional Kinesis Streams, which would drive the cost up higher than an SNS/SQS fanout.
        scarface74 8y ago
        5 reads per second - yeah that’s kind of low. At maximum scale up. I am processing 80 messages simultaneously on 8 instances - each instance is running 10 (I/O bound) asynchronous threads. I could push it higher but the database starts screaming. But with Kinesis, it’s true that you can do only 5 reads per second, but each read can have up to 10,000 records. With SQS, you can only get 10 records per call. I would think you could get much higher throughout with Kinesis, you would just have to handle storing your iterator/sequence number per shard somewhere to know where you left off in case of crash. Kinesis is much better for higher throughput and you can always scale up instead of out if you need consistent throughout. It depends on your use case.

nostrebored 8y ago

I think the sqs and sns naming convention is a bit confusing. Realizing that sns should be the typical choice for an event based architecture is weird to people used to working with a message queue

scarface74 8y ago
SNS and SQS works together but serve two different purposes. SNS is for producers. You use SNS to say something happened. You have no guarantee that any consumer actually successfully processed the message and if the consumer is down, that message is lost to them. The only way a consumer can actually directly process a message is by subscribing to via http or lambda. But again, if the consumer errors or is down, you’re out of luck. SQS is a traditional simple queueing mechanism. It has no fanout capability on its own but you do get the traditional queuing functionality. But it doesn’t make sense logically for more than one process (or group of processes doing the same thing) to consume the queue. If you want the traditional fanout, filtering, multiple queues that do different things on the same event/message, you use SNS and SQS together.
- cthalupa 8y ago
  >If you want the traditional fanout, filtering, multiple queues that do different things on the same event/message, you use SNS and SQS together. For this sort of use case, I've generally opted to go with a Kinesis stream, and I'm having a bit of a hard time understanding why a mix of SNS and SQS would be better here.
  - scarface74 8y ago
    For a simple “something happened” with a message and attributes, Kinesis is overkill and not as flexible. You can send an SNS message with attributes and subscribe to that SNS message with any combination of SQS queues, lambda functions, emails, http endpoints, etc and with any of the subscriptions you can design it so that any of the subscriptions only get messages based on attribute conditions. Also with SQS, you get the standard granular retries, dead letter queues etc. Yes with Kinesis you can do shards but it really doesn’t make sense to have more than one process reading messages from the same shard to scale out processing. With SQS, you can autoscale instances to read from the queue based on the queue size or just subscribe the SNS to an SQS queue and then subscribe the SQS queue to a lambda and let AWS work it’s magic.
    - cthalupa 8y ago
      Much appreciated! This actually has me rethinking the architecture on a project I'm working on right now. It looks like Kinesis would be a little bit cheaper at the volume of data I'm looking at... But the SNS/SQS method will let me sidestep some potential future scale-up concerns I had with the 5 reads per second limit without making a Rube Goldberg machine of Kinesis Analytics feeding into additional Kinesis Streams, which would drive the cost up higher than an SNS/SQS fanout.
      - scarface74 8y ago
        5 reads per second - yeah that’s kind of low. At maximum scale up. I am processing 80 messages simultaneously on 8 instances - each instance is running 10 (I/O bound) asynchronous threads. I could push it higher but the database starts screaming. But with Kinesis, it’s true that you can do only 5 reads per second, but each read can have up to 10,000 records. With SQS, you can only get 10 records per call. I would think you could get much higher throughout with Kinesis, you would just have to handle storing your iterator/sequence number per shard somewhere to know where you left off in case of crash. Kinesis is much better for higher throughput and you can always scale up instead of out if you need consistent throughout. It depends on your use case.