5 ms·

I think there is a useful distinction here though -- you're by definition doing processing there on a "non-exactly-once delivery" system in order to get your "e

by chias 4y ago

I think there is a useful distinction here though -- you're by definition doing processing there on a "non-exactly-once delivery" system in order to get your "exactly-once" result, and by definition anything that requires exactly-once message delivery must do this process itself: it cannot abstracted away into a separate system. So, in essence, you can never have "I will get this exactly once", and at best you can only ever have "I will have a plan for what to do if I get this more than once".

chacham15 4y ago
> So, in essence, you can never have "I will get this exactly once", and at best you can only ever have "I will have a plan for what to do if I get this more than once". I dont understand this at all. TCP has exactly once message delivery that the application layer is completely unaware of...
- majormajor 4y ago
  TCP isn't guaranteed delivery if the receiver crashes; that's what queues and similar persistent systems are introduced to solve, but at-least-once is what you normally get outside of specialized two-phase (write then commit later) idempotent (so writing a second time is OK if the first one was never committed since your producer died and restarted from an earlier position, say) systems, AFAIK.
  - dagss 4y ago
    In theory you could simply wait with TCP ACK until you had committed to your ACID database and then TCP would in fact be exactly once. I guess hardly anyone does this though.
still_grokking 4y ago
Looks like pure philosophical distinction. The question here would be who "you" are. Are "you" the low-level system processing raw messages, or are "you" a system on top of that? The high-level system can rightfully claim to that it receives messages "exactly once"—from the low-level system.
marcosdumay 4y ago
This is something I don't understand with the common idiom. You need some processing for anything on the network, either you accept it or not. Yet, network protocols are described by the behavior they export to their consumers, not by their internals. Well, with the single exception of exactly-once delivery.
xg15 4y ago
> it cannot abstracted away into a separate system. It absolutely can. That's the entire point of TCP.
- pixl97 4y ago
  At the same time, don't trust any network middleware not to break your expectations on TCP behavior... also don't trust that middleware will be visible and you'll know what's going wrong.
- andrewaylett 4y ago
  TCP can't do exactly once in every scenario without fail. It does a pretty good job, but failure in the edge cases is the whole point of the difficulty we're looking at. If the network loses the last packet in a TCP stream, then goes down for an indefinite period, the two ends of the connection have irreconcilable differences of understanding about whether the entire transmission was received. As far as the recipient is concerned, they have an entire message that's fully ack'd and so they should process it. As far as the sender is concerned, they have no ack for the last chunk of data and must redeliver it. This is the crux of the Byzantine Generals problem. TCP can solve problems that happen in its own domain, and give reliable in-order delivery once over an unreliable network up to a point. It's not able to provide exactly once semantics in all scenarios though. Because that's logically impossible.

by chias 4y ago

chacham15 4y ago

> So, in essence, you can never have "I will get this exactly once", and at best you can only ever have "I will have a plan for what to do if I get this more than once". I dont understand this at all. TCP has exactly once message delivery that the application layer is completely unaware of...

majormajor 4y ago
TCP isn't guaranteed delivery if the receiver crashes; that's what queues and similar persistent systems are introduced to solve, but at-least-once is what you normally get outside of specialized two-phase (write then commit later) idempotent (so writing a second time is OK if the first one was never committed since your producer died and restarted from an earlier position, say) systems, AFAIK.
- dagss 4y ago
  In theory you could simply wait with TCP ACK until you had committed to your ACID database and then TCP would in fact be exactly once. I guess hardly anyone does this though.

still_grokking 4y ago

Looks like pure philosophical distinction. The question here would be who "you" are. Are "you" the low-level system processing raw messages, or are "you" a system on top of that? The high-level system can rightfully claim to that it receives messages "exactly once"—from the low-level system.

marcosdumay 4y ago

This is something I don't understand with the common idiom. You need some processing for anything on the network, either you accept it or not. Yet, network protocols are described by the behavior they export to their consumers, not by their internals. Well, with the single exception of exactly-once delivery.

xg15 4y ago

> it cannot abstracted away into a separate system. It absolutely can. That's the entire point of TCP.

pixl97 4y ago
At the same time, don't trust any network middleware not to break your expectations on TCP behavior... also don't trust that middleware will be visible and you'll know what's going wrong.
andrewaylett 4y ago
TCP can't do exactly once in every scenario without fail. It does a pretty good job, but failure in the edge cases is the whole point of the difficulty we're looking at. If the network loses the last packet in a TCP stream, then goes down for an indefinite period, the two ends of the connection have irreconcilable differences of understanding about whether the entire transmission was received. As far as the recipient is concerned, they have an entire message that's fully ack'd and so they should process it. As far as the sender is concerned, they have no ack for the last chunk of data and must redeliver it. This is the crux of the Byzantine Generals problem. TCP can solve problems that happen in its own domain, and give reliable in-order delivery once over an unreliable network up to a point. It's not able to provide exactly once semantics in all scenarios though. Because that's logically impossible.