6 ms·

> give me a queue with FIFO and exactly once delivery https://bravenewgeek.com/you-cannot-have-exactly-once-delivery/ https://bravenewgeek.com/you-cannot-have-

• scarface74 8y ago
  Done https://aws.amazon.com/about-aws/whats-new/2016/11/amazon-sqs-introduces-fifo-queues-with-exactly-once-processing-and-lower-prices-for-standard-queues/ https://aws.amazon.com/about-aws/whats-new/2016/11/amazon-sq...
  • dagenix 8y ago
    Well, not quite. What SQS actually offers is: exactly once delivery, except when the message is delivered multiple times. Quoting from the SQS docs [1]: > The visibility timeout begins when Amazon SQS returns a message. During this time, the consumer processes and deletes the message. However, if the consumer fails before deleting the message and your system doesn't call the DeleteMessage action for that message before the visibility timeout expires, the message becomes visible to other consumers and the message is received again. If a message must be received only once, your consumer should delete it within the duration of the visibility timeout. If a consumer receives a message and then loses its network connection to SQS, the visibility timeout will expire before it can delete the message and it will be delivered a second time to another consumer. I would very much suggest reading through the linked article by GP and maybe more about the Byzantine Generals Problem which establishes why exactly once delivery isn't just hard, but impossible. And Amazon's marketing department can't change that. [1] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html#configuring-visibility-timeout https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
    • deleted 8y ago
      [deleted]
    • deleted 8y ago
      [deleted]
    • scarface74 8y ago
      Or you could post the quote from the link I posted. You can now use Amazon Simple Queue Service (SQS) for applications that require messages to be processed in a strict sequence and exactly once using First-in, First-out (FIFO) queues. FIFO queues are designed to ensure that the order in which messages are sent and received is strictly preserved and that each message is processed exactly once
      • dagenix 8y ago
        What exactly is it about: > However, if the consumer fails before deleting the message and your system doesn't call the DeleteMessage action for that message before the visibility timeout expires, the message becomes visible to other consumers and the message is received again. that you fail to understand. The phrase "received again" means that the message is delivered more than once. More than once is more than exactly once. The quote you are referencing is marketing nonsense.
        • scarface74 8y ago
          If you want the message to be only received once regardless of whether it was successful processed, you set the MaxReceive count to 1 and set up a dead letter queue as part of the redrive policy and you use fifo queues instead of the standard queue. It isn’t “marketing nonsense”. It’s more just quoting from one site on the Internet, instead of understanding the system as a whole.
          • orf 8y ago
            That would make it at most once delivery.
            • scarface74 8y ago
              The fact that a process did a GetMessage means it was delivered at least once.
              • 8y ago
  • orf 8y ago
    Ah yes, that age old showdown: the fundamentals of distributed systems vs a shiny marketing page. It doesn’t make what they say any more true. SQS is at least once delivery, and the fact you think otherwise is down to great marketing and maybe a bit of Dunning–Kruger effect.
    • scarface74 8y ago
      And I posted above how you can ensure “at most” once - send it to a dead letter queue by setting the retrycount to 1... Again, understanding how things work takes more than just reading a single web page.
      • asark 8y ago
        > exactly once delivery > done > SQS is at least once delivery > And I posted above how you can ensure “at most” once So, uh, what happened to exactly once?
        • scarface74 8y ago
          SQS is only meant for multiple publishers and a single subscriber or set of subscribers that do the same thing. A message in an SQS queue is either delivered or in the queue. SQS operates on a strictly polling model. Meaning you’re not going to put a message directly in an SQS queue and have it processed by multiple types of subscribers. Are you expecting some generic system to make sure your subscribers are up and running and polling? Better question is, do you know what SQS is and have you ever done anything with it? This is one definition of “exactly once”. https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ https://www.confluent.io/blog/exactly-once-semantics-are-pos... Exactly once semantics: even if a producer retries sending a message, it leads to the message being delivered exactly once to the end consumer. This is done with AWS FIFO queues by: Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue. So exactly once is handle by the producer, the queueing service, and the subscriber.
• 013a 8y ago
  Apologies, I misspoke. SQS does have a FIFO+Exactly Once Processing mode, so it will never introduce duplicate messages into the queue. It of course doesn't have exactly once delivery without additional state tracking on the consumer outside of the queue.