4 ms·

4. getMessage called; message sent; consumer experiences critical hardware failure before it gets through message; visibility timeout expires; message goes to D

4. getMessage called; message sent; consumer experiences critical hardware failure before it gets through message; visibility timeout expires; message goes to DLQ. In this case, the message is never delivered.

scarface74 8y ago
If the API was called by the consumer and the connection was alive long enough for the request to be sent without the connection closing, the message was sent to the client. If the client failed halfway through receiving the message the server would have received a connection closed event. That’s basic TCP/IP.
- orf 8y ago
  The point that a fair few people here have been trying to make is that things are not as simple as you are quoting from the documentation. There are a variety of networking, hardware and software issues that can conspire to mean that it’s impossible to guarantee exactly once delivery in general. Every networking protocol is built precisely around this integral fact. Just because it’s at a higher level doesn’t mean the underlying issues are not still there. Heck, we cannot even guarantee it in real life. Granted, AWS likely has a pretty specialized networking stack and I doubt network partitions are the norm, but that doesn’t really matter. For some degree of once, and some degree of exactly, it’s possible. Which is what is happening here.
  - scarface74 8y ago
    But you have failed to come up with a scenario outside of the three that I named and the fourth one which you posted is a type of one those - the message was received and not processed - ie the consumer crashed after from SQS’s perspective, the message was received. AWS never guaranteed that your process would work correctly. It guarantees that if you as a producer either sent a message with a uniqueid or unique content within five minutes, it wouldn’t enqueue the message more than once. It also guarantees that if your process acknowledged receipt of a message, and you set up a DLQ. It wouldn’t send the same message more then once. No AWS can’t promise that your consumer is up and running and polling for a message. The best they can do is allow you to setup an alert to tell you that a message has been waiting for you to process it for more than “approximately” x seconds.
    - orf 8y ago
      Exactly once processing, not exactly once delivery. Because that's impossible. Provably so. https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem
    - dagenix 8y ago
      > But you have failed to come up with a scenario outside of the three that I named and the fourth one which you posted is a type of one those - the message was received and not processed - ie the consumer crashed after from SQS’s perspective, the message was received. AWS never guaranteed that your process would work correctly. What it sounds like you are saying is that as soon as SQS sends the message to its consumer, then SQS can consider that a successful delivery since it isn't responsible for anything that happens next. The problem is, while such a metric would support the AWS marketing, it's not really useful in the real world nor is it the tradition definition for what "exactly once delivery" means. In order to be "exactly once delivery", it needs to guarantee that a message actually makes it to a consumer, given an unreliable communication channel. The consumer is an important part of the system. I think my example of a server failing may have confused things. So, let's limit this to just network failures. Also, let's look at TCP from the packet level. Once you get down to this level, TCP really starts to look a lot like UDP. Reading https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem is handy - it's a thought experiment, but, what it proves is that it's not possible to perfectly coordinate state across an unreliable network. Why that is important is because it means that once you send a packet of information, its not possible to always know if it was received by the other end - 1) it might have been lost in transit; 2) it might have made it, but the receiver sent an acknowledgment message that got lost; 3) it made it, but the acknowledgment message has been delayed and hasn't been received yet. Whats interesting about the Two Generals Problem is that it shows that the best we can do, is to reduce the chance of lost messages interfering in communication, but we can't eliminate it. So, we have a few scenarios we can work through, packet-by-packet: 1. If everything goes right: a) consumer sends a getMessage packet b) SQS receives a getMessage packet c) SQS sends a Message packet d) consumer receives a Message packet e) consumer processes the message f) consumer send a deleteMessage packet g) SQS receives a deleteMessage packet and deletes the message However, the network is unreliable. So, at any point, SQS and the consumer can stop being able to communicate for arbitrarily long chunks of time. So, we can end up with something like: 2. With a DLQ setup, if the network drops at the right time, we lose a message: a) consumer sends a getMessage packet b) SQS receives a getMessage packet c) SQS sends a Message packet d) network drops for the duration of the visibility period e1) SQS send messages to DLQ e2) consumer retries getMessage call - gets a different message 3. Without a DLQ setup, if the network drops at the right time, we double-deliver a message: a) consumer sends a getMessage packet b) SQS receives a getMessage packet c) SQS sends a Message packet d) consumer receives a Message packet e) consumer processes the message f) network drops for the duration of the visibility period g) SQS sees that the message hasn't been deleted within the visibility period; message becomes visible again h) another consumer sends a getMessage pack i) SQS sends a Message packet with the same message as before You might say something along the lines of: Let's add some extra acknowledgment packets in. But, this is where the Two Generals problem rears its ugly head - it doesn't matter how many acknowledgments we add in, we're still faced with fundamental limitations of agreeing on state over an unreliable channel. And if you extend that, it implies that its not possible for both ends of a TCP connection to agree on if a connection ended cleanly or ended with a timeout - and the distinction is important if, as you've suggested, you want to use TCP connection failures as part of determining if a message was delivered. I've never used SQS. It's probably fine. Maybe its great, I dunno. Honestly, the features it have sound pretty useful for guaranteeing "usually once delivery". But, they can't guarantee "exactly once delivery" - doing that isn't just hard, it's impossible, it's been proved impossible, and marketing can't change that.

scarface74 8y ago

If the API was called by the consumer and the connection was alive long enough for the request to be sent without the connection closing, the message was sent to the client. If the client failed halfway through receiving the message the server would have received a connection closed event. That’s basic TCP/IP.

orf 8y ago
The point that a fair few people here have been trying to make is that things are not as simple as you are quoting from the documentation. There are a variety of networking, hardware and software issues that can conspire to mean that it’s impossible to guarantee exactly once delivery in general. Every networking protocol is built precisely around this integral fact. Just because it’s at a higher level doesn’t mean the underlying issues are not still there. Heck, we cannot even guarantee it in real life. Granted, AWS likely has a pretty specialized networking stack and I doubt network partitions are the norm, but that doesn’t really matter. For some degree of once, and some degree of exactly, it’s possible. Which is what is happening here.
- scarface74 8y ago
  But you have failed to come up with a scenario outside of the three that I named and the fourth one which you posted is a type of one those - the message was received and not processed - ie the consumer crashed after from SQS’s perspective, the message was received. AWS never guaranteed that your process would work correctly. It guarantees that if you as a producer either sent a message with a uniqueid or unique content within five minutes, it wouldn’t enqueue the message more than once. It also guarantees that if your process acknowledged receipt of a message, and you set up a DLQ. It wouldn’t send the same message more then once. No AWS can’t promise that your consumer is up and running and polling for a message. The best they can do is allow you to setup an alert to tell you that a message has been waiting for you to process it for more than “approximately” x seconds.
  - orf 8y ago
    Exactly once processing, not exactly once delivery. Because that's impossible. Provably so. https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem
  - dagenix 8y ago
    > But you have failed to come up with a scenario outside of the three that I named and the fourth one which you posted is a type of one those - the message was received and not processed - ie the consumer crashed after from SQS’s perspective, the message was received. AWS never guaranteed that your process would work correctly. What it sounds like you are saying is that as soon as SQS sends the message to its consumer, then SQS can consider that a successful delivery since it isn't responsible for anything that happens next. The problem is, while such a metric would support the AWS marketing, it's not really useful in the real world nor is it the tradition definition for what "exactly once delivery" means. In order to be "exactly once delivery", it needs to guarantee that a message actually makes it to a consumer, given an unreliable communication channel. The consumer is an important part of the system. I think my example of a server failing may have confused things. So, let's limit this to just network failures. Also, let's look at TCP from the packet level. Once you get down to this level, TCP really starts to look a lot like UDP. Reading https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem is handy - it's a thought experiment, but, what it proves is that it's not possible to perfectly coordinate state across an unreliable network. Why that is important is because it means that once you send a packet of information, its not possible to always know if it was received by the other end - 1) it might have been lost in transit; 2) it might have made it, but the receiver sent an acknowledgment message that got lost; 3) it made it, but the acknowledgment message has been delayed and hasn't been received yet. Whats interesting about the Two Generals Problem is that it shows that the best we can do, is to reduce the chance of lost messages interfering in communication, but we can't eliminate it. So, we have a few scenarios we can work through, packet-by-packet: 1. If everything goes right: a) consumer sends a getMessage packet b) SQS receives a getMessage packet c) SQS sends a Message packet d) consumer receives a Message packet e) consumer processes the message f) consumer send a deleteMessage packet g) SQS receives a deleteMessage packet and deletes the message However, the network is unreliable. So, at any point, SQS and the consumer can stop being able to communicate for arbitrarily long chunks of time. So, we can end up with something like: 2. With a DLQ setup, if the network drops at the right time, we lose a message: a) consumer sends a getMessage packet b) SQS receives a getMessage packet c) SQS sends a Message packet d) network drops for the duration of the visibility period e1) SQS send messages to DLQ e2) consumer retries getMessage call - gets a different message 3. Without a DLQ setup, if the network drops at the right time, we double-deliver a message: a) consumer sends a getMessage packet b) SQS receives a getMessage packet c) SQS sends a Message packet d) consumer receives a Message packet e) consumer processes the message f) network drops for the duration of the visibility period g) SQS sees that the message hasn't been deleted within the visibility period; message becomes visible again h) another consumer sends a getMessage pack i) SQS sends a Message packet with the same message as before You might say something along the lines of: Let's add some extra acknowledgment packets in. But, this is where the Two Generals problem rears its ugly head - it doesn't matter how many acknowledgments we add in, we're still faced with fundamental limitations of agreeing on state over an unreliable channel. And if you extend that, it implies that its not possible for both ends of a TCP connection to agree on if a connection ended cleanly or ended with a timeout - and the distinction is important if, as you've suggested, you want to use TCP connection failures as part of determining if a message was delivered. I've never used SQS. It's probably fine. Maybe its great, I dunno. Honestly, the features it have sound pretty useful for guaranteeing "usually once delivery". But, they can't guarantee "exactly once delivery" - doing that isn't just hard, it's impossible, it's been proved impossible, and marketing can't change that.