4 ms·

That seems like a distinction without a difference to me. Why should I care if the thing I get exactly one of is called "processing" or "delivery"?

by lisper 2y ago

jerf 2y ago
Because you need to understand that your processing code is constrained by the fact that you can't get exactly-once delivery. You must write your processing code to handle it one way or another. There's some libraries that try to wrap the abstraction of processing exactly once around the code, but those libraries still impose constraints on the sort of code you can write. They can make it easier but they can't fully remove the need to think about how your processing code works. It isn't exactly the same. This is why people like me insist it's important to understand that you can not have exactly-once delivery. There is no library that can make that just go away, they can only shuffle around exactly where the lumps under the carpet live, and if one programs with the mistaken idea that these libraries really do solve "exactly once" delivery, one will get into deep trouble. Possibly the "my architecture is fundamentally broken and can't be rescued" sort of trouble.
- lisper 2y ago
  > Because you need to understand that your processing code is constrained by the fact that you can't get exactly-once delivery. Why do I need to understand that? Why can I not put an abstraction layer that provides me with the illusion of exactly-once delivery? > There is no library that can make that just go away Well, this is the thing that I dispute. I believe that there is a library I can write to make it go away if I have at-least-once delivery. In fact, I claim that writing such a library is an elementary exercise. The TCP protocol is an existence proof. Where is the flaw in my reasoning?
  - pton_xd 2y ago
    > Why can I not put an abstraction layer that provides me with the illusion of exactly-once delivery? You can do that. You can implement a video conferencing system ontop of TCP, and it will even work, technically. It will just have terrible performance characteristics that you'll never be able to fix. You might even call it fundamentally broken.
    - lisper 2y ago
      OK, I don't dispute that, but that is a very different claim than "you can't get exactly-once delivery." You can (if you have at-least-once delivery).
      - sethammons 2y ago
        Read the two general's problem. It is proven that exactly once _delivery_ is a physical and mathematical impossibility. Like you said, you can simulate it with exactly once _processing_. But to do that, you have to know to do that. Others are saying "you can't divide by zero" and you are saying "yeah, but if I detect a zero and the do something different, then it is the same thing." No, knowing you have to do something is the very point of acknowledging you can't divide by zero. Because you can't have exactly once delivery, you have to deal with it. One trick is duplicate checks or idempotent writes. This gives exactly once processing. This also takes additional overhead and why audio and video stream processing doesn't typically do the additional checks. I have fixed many bugs written by people who believe the network is reliable. I even hired one when during the interview and we talked about this kind of issue that they realized why they were getting duplicated writes reading from sqs or sns at $existing_job. They were green but smart and she became a real asset to the team.
        lisper 2y ago
        What makes you thin the 2GP is relevant here? 2GP has to do with coordination and consensus, not exactly-once-delivery. > you can simulate it with exactly once _processing_ What exactly do you think is the difference between "simulating" exactly-once delivery and actually having exactly-once delivery? What do you think "delivery" means in the context of a computer network?
  - shepherdjerred 2y ago
    TCP is at least once, not exactly once. Here's another useful article: https://blog.bulloak.io/post/20200917-the-impossibility-of-exactly-once/ https://blog.bulloak.io/post/20200917-the-impossibility-of-e... TCP does not solve the two generals problem. TCP gets around this limitation by requiring only one ACK. https://en.wikipedia.org/wiki/Two_Generals%27_Problem#Engineering_approaches https://en.wikipedia.org/wiki/Two_Generals%27_Problem#Engine... https://www.scaler.com/topics/computer-network/two-generals-problem-in-tcp/ https://www.scaler.com/topics/computer-network/two-generals-...
    - lisper 2y ago
      OK, but you have to do a little extrapolating here because the claim is not that you can do exactly-once under all circumstances. That is obviously false because you can't do exactly-once in a situation where all comms are down indefinitely. My claim is that if I have at-least-once then I can build exactly-once out of that.
      - shepherdjerred 2y ago
        This isn't about indefinite communication loss. Obviously no progress is possible in that case. The two generals' problem has nothing to do with a permanent failure. I think there is a lot of literature out there if you're really interested in understanding and I'm happy to provide more links if you'd like.
        lisper 2y ago
        What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery.
        shepherdjerred 2y ago
        The link I posted above explains the connection: https://blog.bulloak.io/post/20200917-the-impossibility-of-exactly-once/ https://blog.bulloak.io/post/20200917-the-impossibility-of-e... Particularly: > The sender cannot know if a message was delivered since transport is unreliable; thus, one or more acknowledgement messages are required. Moreover, the sender cannot distinguish between message delivery errors, acknowledgement delivery errors, or delays (either in processing or because or network unreliability). > The recipient is forced to send the acknowledgement only after the message is either processed (or persisted for processing) because an acknowledgement before processing would not work: if the recipient exhibits a fault before processing that would cause the loss of the message. > In the case that that acknowledgement is lost, the sender can’t know (due to lack of insight in the recipient’s state) whether the recipient failed before scheduling the message for processing (in essence losing the message) or if the recipient is just running a bit slow, or if the acknowledgement message was lost. Now, if the sender decides to re-deliver, then the recipient may end up receiving the message twice if the acknowledgement was dropped (for example). On the other hand, if the sender decides to not re-deliver, then the recipient may end up not processing the message at all if the issue was that the message wasn’t scheduled for processing.
  - Bognar 2y ago
    TCP implementations are an abstraction that work 99.99% of the time, but are still vulnerable to two generals when you look close. TCP is implemented in the kernel with a buffer, the kernel responds with ACKs before an application reads the data. There is no guarantee that the application reads from that buffer (e.g. the process could crash), so the client on the other end believes that the application has received the message even though it hasn't. The kernel is handling at-least-once delivery with the network boundary and turning it into at-most-once with the process boundary.
    - lisper 2y ago
      > still vulnerable to two generals What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery. > TCP is implemented in the kernel with a buffer, the kernel responds with ACKs before an application reads the data. True. Why do you think that matters? > There is no guarantee that the application reads from that buffer So? What does that have to do with exactly-once delivery? Even if the application does read the data, there's no guarantee that it does anything with it afterwards. > The kernel is handling at-least-once delivery with the network boundary and turning it into at-most-once with the process boundary. OK, if that's how you're defining your terms, I agree that you cannot have exactly-once delivery. But that makes it a vacuous observation. You can always get at-most-once delivery by disconnecting the network entirely. That provides a 100% guarantee that no message will be delivered more than once (because no message will ever be delivered at all). But that doesn't seem very useful.
shepherdjerred 2y ago
I don't know enough to answer this. I'm sure there is plenty of writing on this subject from people more qualified than me.

That seems like a distinction without a difference to me. Why should I care if the thing I get exactly one of is called "processing" or "delivery"?

by lisper 2y ago

That seems like a distinction without a difference to me. Why should I care if the thing I get exactly one of is called "processing" or "delivery"?

jerf 2y ago

Because you need to understand that your processing code is constrained by the fact that you can't get exactly-once delivery. You must write your processing code to handle it one way or another. There's some libraries that try to wrap the abstraction of processing exactly once around the code, but those libraries still impose constraints on the sort of code you can write. They can make it easier but they can't fully remove the need to think about how your processing code works. It isn't exactly the same. This is why people like me insist it's important to understand that you can not have exactly-once delivery. There is no library that can make that just go away, they can only shuffle around exactly where the lumps under the carpet live, and if one programs with the mistaken idea that these libraries really do solve "exactly once" delivery, one will get into deep trouble. Possibly the "my architecture is fundamentally broken and can't be rescued" sort of trouble.

lisper 2y ago
> Because you need to understand that your processing code is constrained by the fact that you can't get exactly-once delivery. Why do I need to understand that? Why can I not put an abstraction layer that provides me with the illusion of exactly-once delivery? > There is no library that can make that just go away Well, this is the thing that I dispute. I believe that there is a library I can write to make it go away if I have at-least-once delivery. In fact, I claim that writing such a library is an elementary exercise. The TCP protocol is an existence proof. Where is the flaw in my reasoning?
- pton_xd 2y ago
  > Why can I not put an abstraction layer that provides me with the illusion of exactly-once delivery? You can do that. You can implement a video conferencing system ontop of TCP, and it will even work, technically. It will just have terrible performance characteristics that you'll never be able to fix. You might even call it fundamentally broken.
  - lisper 2y ago
    OK, I don't dispute that, but that is a very different claim than "you can't get exactly-once delivery." You can (if you have at-least-once delivery).
    - sethammons 2y ago
      Read the two general's problem. It is proven that exactly once _delivery_ is a physical and mathematical impossibility. Like you said, you can simulate it with exactly once _processing_. But to do that, you have to know to do that. Others are saying "you can't divide by zero" and you are saying "yeah, but if I detect a zero and the do something different, then it is the same thing." No, knowing you have to do something is the very point of acknowledging you can't divide by zero. Because you can't have exactly once delivery, you have to deal with it. One trick is duplicate checks or idempotent writes. This gives exactly once processing. This also takes additional overhead and why audio and video stream processing doesn't typically do the additional checks. I have fixed many bugs written by people who believe the network is reliable. I even hired one when during the interview and we talked about this kind of issue that they realized why they were getting duplicated writes reading from sqs or sns at $existing_job. They were green but smart and she became a real asset to the team.
      - lisper 2y ago
        What makes you thin the 2GP is relevant here? 2GP has to do with coordination and consensus, not exactly-once-delivery. > you can simulate it with exactly once _processing_ What exactly do you think is the difference between "simulating" exactly-once delivery and actually having exactly-once delivery? What do you think "delivery" means in the context of a computer network?
- shepherdjerred 2y ago
  TCP is at least once, not exactly once. Here's another useful article: https://blog.bulloak.io/post/20200917-the-impossibility-of-exactly-once/ https://blog.bulloak.io/post/20200917-the-impossibility-of-e... TCP does not solve the two generals problem. TCP gets around this limitation by requiring only one ACK. https://en.wikipedia.org/wiki/Two_Generals%27_Problem#Engineering_approaches https://en.wikipedia.org/wiki/Two_Generals%27_Problem#Engine... https://www.scaler.com/topics/computer-network/two-generals-problem-in-tcp/ https://www.scaler.com/topics/computer-network/two-generals-...
  - lisper 2y ago
    OK, but you have to do a little extrapolating here because the claim is not that you can do exactly-once under all circumstances. That is obviously false because you can't do exactly-once in a situation where all comms are down indefinitely. My claim is that if I have at-least-once then I can build exactly-once out of that.
    - shepherdjerred 2y ago
      This isn't about indefinite communication loss. Obviously no progress is possible in that case. The two generals' problem has nothing to do with a permanent failure. I think there is a lot of literature out there if you're really interested in understanding and I'm happy to provide more links if you'd like.
      - lisper 2y ago
        What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery.
        shepherdjerred 2y ago
        The link I posted above explains the connection: https://blog.bulloak.io/post/20200917-the-impossibility-of-exactly-once/ https://blog.bulloak.io/post/20200917-the-impossibility-of-e... Particularly: > The sender cannot know if a message was delivered since transport is unreliable; thus, one or more acknowledgement messages are required. Moreover, the sender cannot distinguish between message delivery errors, acknowledgement delivery errors, or delays (either in processing or because or network unreliability). > The recipient is forced to send the acknowledgement only after the message is either processed (or persisted for processing) because an acknowledgement before processing would not work: if the recipient exhibits a fault before processing that would cause the loss of the message. > In the case that that acknowledgement is lost, the sender can’t know (due to lack of insight in the recipient’s state) whether the recipient failed before scheduling the message for processing (in essence losing the message) or if the recipient is just running a bit slow, or if the acknowledgement message was lost. Now, if the sender decides to re-deliver, then the recipient may end up receiving the message twice if the acknowledgement was dropped (for example). On the other hand, if the sender decides to not re-deliver, then the recipient may end up not processing the message at all if the issue was that the message wasn’t scheduled for processing.
- Bognar 2y ago
  TCP implementations are an abstraction that work 99.99% of the time, but are still vulnerable to two generals when you look close. TCP is implemented in the kernel with a buffer, the kernel responds with ACKs before an application reads the data. There is no guarantee that the application reads from that buffer (e.g. the process could crash), so the client on the other end believes that the application has received the message even though it hasn't. The kernel is handling at-least-once delivery with the network boundary and turning it into at-most-once with the process boundary.
  - lisper 2y ago
    > still vulnerable to two generals What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery. > TCP is implemented in the kernel with a buffer, the kernel responds with ACKs before an application reads the data. True. Why do you think that matters? > There is no guarantee that the application reads from that buffer So? What does that have to do with exactly-once delivery? Even if the application does read the data, there's no guarantee that it does anything with it afterwards. > The kernel is handling at-least-once delivery with the network boundary and turning it into at-most-once with the process boundary. OK, if that's how you're defining your terms, I agree that you cannot have exactly-once delivery. But that makes it a vacuous observation. You can always get at-most-once delivery by disconnecting the network entirely. That provides a 100% guarantee that no message will be delivered more than once (because no message will ever be delivered at all). But that doesn't seem very useful.

shepherdjerred 2y ago

I don't know enough to answer this. I'm sure there is plenty of writing on this subject from people more qualified than me.