4 ms·

Well pretty confused. You can't get exactly once delivery using proposed solution in general case because you can not have infinite memory for dedupe.

• lisper 2y ago
  You apparently missed this part: "Just for the sake of completeness I should point out that removing duplicates at the receiver is a pretty extreme oversimplification of what you would do in practice to provide exactly-once delivery. A complete solution would almost certainly be an example of Greenspun's Tenth Law applied to the TCP protocol rather than Common Lisp."
  • two_handfuls 2y ago
    Can you explain this point? I think you are trying to say the application would include an implementation of TCP? I don't see how this is related to memory.
    • lisper 2y ago
      I was responding to this: > You can't get exactly once delivery using proposed solution in general case because you can not have infinite memory for dedupe. That is a specific criticism of a specific and highly oversimplified algorithm for de-duplication that I described purely for illustrative purposes, not as a suggestion for how de-duplication should actually be implemented. Actual implementation of de-duplication is much more complicated, but a solved problem that I didn't think I needed to rehash. A real implementation doesn't require infinite memory. (Actually, even the naive algorithm doesn't require infinite memory, just unbounded memory. Those are not the same.)
      • qaq 2y ago
        You certainly do need to rehash it or provide a link to it being solved without constraints on time window, msgs rates and msgs sizes. You also most certainly want to build it and provide it as service because no existing product claims this capability in unbounded case.
        • lisper 2y ago
          > solved without constraints on time window, msgs rates and msgs sizes But that is not what I am claiming.
          • two_handfuls 2y ago
            The question was for the general case, so yes that is exactly what you are claiming in this thread. Here is the question again: "Well pretty confused. You can't get exactly once delivery using proposed solution in general case because you can not have infinite memory for dedupe."
            • lisper 2y ago
              Only a naive implementation requires infinite memory. (And even then it's unbounded, not infinite.)
              • qaq 2y ago
                You are going in circles so you claim you do have a solution for general case. It would be interesting to learn about details of such solution beyond it has different constraints compared to naive solution.
                • two_handfuls 2y ago
                  Don't bother, it's clear we have a "doesn't fit in the margin" situation here.
                  • lisper 2y ago
                    It's true that a complete solution doesn't fit easily into an HN comment. But one possibility is to send only one message at a time, and keep re-sending it until you receive an acknowledgement. Then send the next message. On the receiving side, deliver the first instance of the multiple copies you receive and discard the rest. (Send acknowledgements for all received instances of course.) This guarantees not only exactly-once delivery with only constant storage required at the receiver, but also in-order delivery. It's not particularly efficient, and it's not what you would want to do in real life, but it would work.
• deleted 2y ago
  [deleted]