5 ms·

>SQS has a many-to-one relationship. You can send messages to a queue from many different producers but only one consumer can be defined. A consumer is another

>SQS has a many-to-one relationship. You can send messages to a queue from many different producers but only one consumer can be defined. A consumer is another application, most often some compute instances such as Lambda, EC2, or Fargate. My understanding is that you can have multiple consumers of an SQS through the use of visibility timeouts[0]. Once a message is consumed it is as if that message doesn't exist for all other consumers until it reaches a timeout period or is marked done by that consumer. You can also manually mark a message as being ready for other consumers. This moves the message back into the queue for the other consumers to see. I'm going to be linking this article to my team. We've been talking about moving to SNS/SQS/etc. and this article helps understand the use cases and distinctions better. [0]: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

specialp 4y ago
I would still view it as many to one. Visibility timeouts are for concurrency. Semantically speaking I would consider one consumer with n concurrent workers as one consumer function/service. In workflow terms, a SNS is a fanout, and SQS is a queue.
- andrew_ 4y ago
  You can subscribe multiple lambdas to an SQS queue. It's not recommended, but it's doable. many-to-many or many-to-one depends on your choices of infrastructure.
  - elsurudo 4y ago
    Yes, but only one of them will receive the message.
    - andrew_ 4y ago
      Only one would receive a particular message, but all will receive messages.
      - elsurudo 4y ago
        That's true, but I wouldn't say that fits the definition of 1:N messaging. Just as the name says, it's a queue.
  - specialp 4y ago
    I guess we can debate the semantics of it because it is technically possible. But it is terrible design to have a SQS to many different consumers. If someone did that I would reject it on review. In any proper usage of SQS it is many to one.
    - andrew_ 4y ago
      there are edge cases where that's desirable. I won't enumerate them here but they're discoverable on the Googles. I also would advise against that kind of passionate adherence to infrastructure dogma, taking a more analytical approach to review.
      - specialp 4y ago
        If you could name even one I would remove the "dogma". I cannot think of why anyone would want to do that. And if someone did want to do that they would have to have a very compelling reason to complicate what is usually an easy thing (One action listening on a queue)
    - afandian 4y ago
      How do you scale out processing if you can't have more than one consumer?
      - specialp 4y ago
        I am not referring to multiple homogeneous consumers processing a queue. That is fine. You have a pool of consumers that can pick up from the queue. That is still considered one entity/actor. The people here are proposing having multiple heterogeneous consumers consume from the queue. That is bad.
sparsely 4y ago
Yes, and you can group the messages such that messages within a group are (almost?) always consumed in order. I think the distinction though is that with SNS each message is consumed by each consumer, whereas SQS each message is consumed by one node (so you can only really have one system that reads from the queue)
015a 4y ago
This is a bad idea. 1. The main point of the visibility timeout is to handle failure. A message is read by a consumer; the visibility timeout starts; that consumer finishes some processing; then deletes the message from the queue. But, what happens if the consumer encounters a fault during processing which destroys its ability to even tell the queue it encountered a fault? The visibility timeout protects against that; the message just naturally reappears in the queue for processing by another consumer. If one overloaded the visibility timeout to also mean "other consumers should process this", you'd lose the ability to handle faults. 2. It also screws up deadletter redrive policies, which are primarily based on visibility timeout lapses (in addition to communicated failures). You basically could not reliably put a deadletter redrive on your queue, which again just means, you're protecting against fewer failure modes. 3. There would be natural, avoidable latency in waiting for the visibility timeout on every fan-out, whatever you set it to. 1 second? 100 consumers? That message is just clogging up the queue for over a minute as it gets fanned-out to everyone. 4. Consumer1 eats the first message, then times-out its visibility; its back in the queue; there's no way to ensure that message isn't just processed again by Consumer1 instead of Consumer2! You're basically tossing a coin and hoping that, eventually, Consumer2 gets its turn at the message, all the while having Consumer1 reprocess the message an indefinite number of times. 5. Someone has to delete the message. Who? The "last" component to touch it? Once all the other components are done? How do you coordinate that? Theres no guarantee of ordering on when each component sees the message. You'd need some kind of external state, and at that point, why are you even using SQS? You could theoretically have each consumer read from a queue, process the message, delete that message from the queue, then redrive the message into a new queue for processing by another consumer. This may make sense if you have strict ordering needs for processing but still want the benefits of SQS. You could even have it redrive into N queues for N consumers at the same time. But, at that point, why? We're trying to put a square peg in a round hole; SQS is designed for single consumers. There are far better and simpler tools out there if what you're looking for is multi-consumer fan-out.
- dimitrios1 4y ago
  I have used SQS in the parent's suggested fashion for many years. I feel like your points are overstated. Visibility timeout's "main point" is not to only handle failure nor do the AWS docs themselves state that. AWS's built-in redrive policies have been more than sufficient to correctly handle error scenarios. > there's no way to ensure that message isn't just processed again by Consumer1 instead of Consumer2! Correct, but this isn't the job of the pipe. Smart endpoints, dumb pipes.
atwebb 4y ago
Why not go pub/sub model if you need multiple consumers?

specialp 4y ago

I would still view it as many to one. Visibility timeouts are for concurrency. Semantically speaking I would consider one consumer with n concurrent workers as one consumer function/service. In workflow terms, a SNS is a fanout, and SQS is a queue.

andrew_ 4y ago
You can subscribe multiple lambdas to an SQS queue. It's not recommended, but it's doable. many-to-many or many-to-one depends on your choices of infrastructure.
- elsurudo 4y ago
  Yes, but only one of them will receive the message.
  - andrew_ 4y ago
    Only one would receive a particular message, but all will receive messages.
    - elsurudo 4y ago
      That's true, but I wouldn't say that fits the definition of 1:N messaging. Just as the name says, it's a queue.
- specialp 4y ago
  I guess we can debate the semantics of it because it is technically possible. But it is terrible design to have a SQS to many different consumers. If someone did that I would reject it on review. In any proper usage of SQS it is many to one.
  - andrew_ 4y ago
    there are edge cases where that's desirable. I won't enumerate them here but they're discoverable on the Googles. I also would advise against that kind of passionate adherence to infrastructure dogma, taking a more analytical approach to review.
    - specialp 4y ago
      If you could name even one I would remove the "dogma". I cannot think of why anyone would want to do that. And if someone did want to do that they would have to have a very compelling reason to complicate what is usually an easy thing (One action listening on a queue)
  - afandian 4y ago
    How do you scale out processing if you can't have more than one consumer?
    - specialp 4y ago
      I am not referring to multiple homogeneous consumers processing a queue. That is fine. You have a pool of consumers that can pick up from the queue. That is still considered one entity/actor. The people here are proposing having multiple heterogeneous consumers consume from the queue. That is bad.

sparsely 4y ago

Yes, and you can group the messages such that messages within a group are (almost?) always consumed in order. I think the distinction though is that with SNS each message is consumed by each consumer, whereas SQS each message is consumed by one node (so you can only really have one system that reads from the queue)

015a 4y ago

This is a bad idea. 1. The main point of the visibility timeout is to handle failure. A message is read by a consumer; the visibility timeout starts; that consumer finishes some processing; then deletes the message from the queue. But, what happens if the consumer encounters a fault during processing which destroys its ability to even tell the queue it encountered a fault? The visibility timeout protects against that; the message just naturally reappears in the queue for processing by another consumer. If one overloaded the visibility timeout to also mean "other consumers should process this", you'd lose the ability to handle faults. 2. It also screws up deadletter redrive policies, which are primarily based on visibility timeout lapses (in addition to communicated failures). You basically could not reliably put a deadletter redrive on your queue, which again just means, you're protecting against fewer failure modes. 3. There would be natural, avoidable latency in waiting for the visibility timeout on every fan-out, whatever you set it to. 1 second? 100 consumers? That message is just clogging up the queue for over a minute as it gets fanned-out to everyone. 4. Consumer1 eats the first message, then times-out its visibility; its back in the queue; there's no way to ensure that message isn't just processed again by Consumer1 instead of Consumer2! You're basically tossing a coin and hoping that, eventually, Consumer2 gets its turn at the message, all the while having Consumer1 reprocess the message an indefinite number of times. 5. Someone has to delete the message. Who? The "last" component to touch it? Once all the other components are done? How do you coordinate that? Theres no guarantee of ordering on when each component sees the message. You'd need some kind of external state, and at that point, why are you even using SQS? You could theoretically have each consumer read from a queue, process the message, delete that message from the queue, then redrive the message into a new queue for processing by another consumer. This may make sense if you have strict ordering needs for processing but still want the benefits of SQS. You could even have it redrive into N queues for N consumers at the same time. But, at that point, why? We're trying to put a square peg in a round hole; SQS is designed for single consumers. There are far better and simpler tools out there if what you're looking for is multi-consumer fan-out.

dimitrios1 4y ago
I have used SQS in the parent's suggested fashion for many years. I feel like your points are overstated. Visibility timeout's "main point" is not to only handle failure nor do the AWS docs themselves state that. AWS's built-in redrive policies have been more than sufficient to correctly handle error scenarios. > there's no way to ensure that message isn't just processed again by Consumer1 instead of Consumer2! Correct, but this isn't the job of the pipe. Smart endpoints, dumb pipes.

atwebb 4y ago

Why not go pub/sub model if you need multiple consumers?