3 ms·

That would make it at most once delivery.

by orf 8y ago

scarface74 8y ago
The fact that a process did a GetMessage means it was delivered at least once.
- orf 8y ago
  It does? How? Can you think of any conditions that might make that not true? Like you said several times, knowing what’s going on takes more than just reading several AWS documentation pages. And to boot, that stretches the definition of delivery. You’d hardly consider a letter delivered if the postman left it outside the postbox and it blew away.
  - scarface74 8y ago
    There are really only a few cases when using fifo queues This will happen no matter what: Message enqueued -> getMessage called -> message visibility state set to invisible -> Various conditions: 1. message processed successfully -> message deleted. message delivered exactly once 2. Message failed to process successfully -> process caught error -> invisibility timeout manually set to 0 by process -> message ends up in DLQ exactly once delivery 3. Message processed successfully -> client failed to delete message —> message times out automatically -> message delivered to DLQ exactly once delivery And to boot, that stretches the definition of delivery. You’d hardly consider a letter delivered if the postman left it outside the postbox and it blew away. The message in the failure case didn’t “blow away”. AWS guaranteed exactly once delivery and it put the messages that failed in a dead letter queue. It’s up to the business to decide remediation steps in failure cases. You can even set up an alert when the DLQ is not empty.
    - dagenix 8y ago
      4. getMessage called; message sent; consumer experiences critical hardware failure before it gets through message; visibility timeout expires; message goes to DLQ. In this case, the message is never delivered.
      - scarface74 8y ago
        If the API was called by the consumer and the connection was alive long enough for the request to be sent without the connection closing, the message was sent to the client. If the client failed halfway through receiving the message the server would have received a connection closed event. That’s basic TCP/IP.
        orf 8y ago
        The point that a fair few people here have been trying to make is that things are not as simple as you are quoting from the documentation. There are a variety of networking, hardware and software issues that can conspire to mean that it’s impossible to guarantee exactly once delivery in general. Every networking protocol is built precisely around this integral fact. Just because it’s at a higher level doesn’t mean the underlying issues are not still there. Heck, we cannot even guarantee it in real life. Granted, AWS likely has a pretty specialized networking stack and I doubt network partitions are the norm, but that doesn’t really matter. For some degree of once, and some degree of exactly, it’s possible. Which is what is happening here.
        scarface74 8y ago
        But you have failed to come up with a scenario outside of the three that I named and the fourth one which you posted is a type of one those - the message was received and not processed - ie the consumer crashed after from SQS’s perspective, the message was received. AWS never guaranteed that your process would work correctly. It guarantees that if you as a producer either sent a message with a uniqueid or unique content within five minutes, it wouldn’t enqueue the message more than once. It also guarantees that if your process acknowledged receipt of a message, and you set up a DLQ. It wouldn’t send the same message more then once. No AWS can’t promise that your consumer is up and running and polling for a message. The best they can do is allow you to setup an alert to tell you that a message has been waiting for you to process it for more than “approximately” x seconds.
        orf 8y ago

That would make it at most once delivery.

by orf 8y ago

That would make it at most once delivery.

scarface74 8y ago

The fact that a process did a GetMessage means it was delivered at least once.

orf 8y ago
It does? How? Can you think of any conditions that might make that not true? Like you said several times, knowing what’s going on takes more than just reading several AWS documentation pages. And to boot, that stretches the definition of delivery. You’d hardly consider a letter delivered if the postman left it outside the postbox and it blew away.
- scarface74 8y ago
  There are really only a few cases when using fifo queues This will happen no matter what: Message enqueued -> getMessage called -> message visibility state set to invisible -> Various conditions: 1. message processed successfully -> message deleted. message delivered exactly once 2. Message failed to process successfully -> process caught error -> invisibility timeout manually set to 0 by process -> message ends up in DLQ exactly once delivery 3. Message processed successfully -> client failed to delete message —> message times out automatically -> message delivered to DLQ exactly once delivery And to boot, that stretches the definition of delivery. You’d hardly consider a letter delivered if the postman left it outside the postbox and it blew away. The message in the failure case didn’t “blow away”. AWS guaranteed exactly once delivery and it put the messages that failed in a dead letter queue. It’s up to the business to decide remediation steps in failure cases. You can even set up an alert when the DLQ is not empty.
  - dagenix 8y ago
    4. getMessage called; message sent; consumer experiences critical hardware failure before it gets through message; visibility timeout expires; message goes to DLQ. In this case, the message is never delivered.
    - scarface74 8y ago
      If the API was called by the consumer and the connection was alive long enough for the request to be sent without the connection closing, the message was sent to the client. If the client failed halfway through receiving the message the server would have received a connection closed event. That’s basic TCP/IP.
      - orf 8y ago
        The point that a fair few people here have been trying to make is that things are not as simple as you are quoting from the documentation. There are a variety of networking, hardware and software issues that can conspire to mean that it’s impossible to guarantee exactly once delivery in general. Every networking protocol is built precisely around this integral fact. Just because it’s at a higher level doesn’t mean the underlying issues are not still there. Heck, we cannot even guarantee it in real life. Granted, AWS likely has a pretty specialized networking stack and I doubt network partitions are the norm, but that doesn’t really matter. For some degree of once, and some degree of exactly, it’s possible. Which is what is happening here.
        scarface74 8y ago
        But you have failed to come up with a scenario outside of the three that I named and the fourth one which you posted is a type of one those - the message was received and not processed - ie the consumer crashed after from SQS’s perspective, the message was received. AWS never guaranteed that your process would work correctly. It guarantees that if you as a producer either sent a message with a uniqueid or unique content within five minutes, it wouldn’t enqueue the message more than once. It also guarantees that if your process acknowledged receipt of a message, and you set up a DLQ. It wouldn’t send the same message more then once. No AWS can’t promise that your consumer is up and running and polling for a message. The best they can do is allow you to setup an alert to tell you that a message has been waiting for you to process it for more than “approximately” x seconds.
        orf 8y ago