4 ms·

As GP said: > if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it. Even QUIC has an ACK-like s

As GP said: > if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it. Even QUIC has an ACK-like signal so you will never avoid it as otherwise you will deal with the two generals' problem (https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem). The reason QUIC is more efficient than TCP is simply due to the fact that TCP can't evolve further due to middleboxes (called ossification, https://en.wikipedia.org/wiki/Protocol_ossification https://en.wikipedia.org/wiki/Protocol_ossification), not that TCP is inherently bad.

_8j50 4y ago
Thanks for the explanation, I didn't know that about QUIC. But QUIC is trying to implement reliable delivery for applications, which makes sense as a general purpose protocol. but if I were doing a file transfer app/protocol these days for example I would send hash+Mac+metadata reliably and then check chunks of the file (unreliable+encrypted UDP)as they are decrypted (have to anyways for integrity checking as part if decryption), and peform reliable retransmission request when decryptiom fails. That's what I meant by application layer, application and use-case specific protocols. but perhaps general purpose protocols are still better (I am not sure, I was asking in my post, sorry for not being clear)
- zinekeller 4y ago
  Then you would generally try to look at probabilistic reliability. One way systems like digital broadcasting requires probabilistic reliability so a lot of research was done here. Digital broadcasting has produced a lot of answers in this area, as you need to deal with both data corruption and missing data. On the data corruption side, you would want to implement a forward-error correction (FEC, https://en.wikipedia.org/wiki/Error_correction_code#Forward_error_correction https://en.wikipedia.org/wiki/Error_correction_code#Forward_...) to allow recovery of corrupted packets, while on missing data it can be mitigated by fountain codes (https://en.wikipedia.org/wiki/Fountain_code https://en.wikipedia.org/wiki/Fountain_code) that is basically FEC but deals with missing pieces instead of incorrect data. This of course does not guarantee 100% delivery but in practical terms this is humanly reliable.
  - _8j50 4y ago
    Thank you, that certainly gives me a lot to think about in this area.
- simmervigor 4y ago
  You could build this as an application mapping over QUIC. Within a single QUIC connection use reliable streams for the metadata delivery, and unreliable datagrams [1] for the payload. The benefit of a single connection is shared security context and shared fates of different datatype transfers. [1] - https://www.rfc-editor.org/rfc/rfc9221.html https://www.rfc-editor.org/rfc/rfc9221.html

_8j50 4y ago

Thanks for the explanation, I didn't know that about QUIC. But QUIC is trying to implement reliable delivery for applications, which makes sense as a general purpose protocol. but if I were doing a file transfer app/protocol these days for example I would send hash+Mac+metadata reliably and then check chunks of the file (unreliable+encrypted UDP)as they are decrypted (have to anyways for integrity checking as part if decryption), and peform reliable retransmission request when decryptiom fails. That's what I meant by application layer, application and use-case specific protocols. but perhaps general purpose protocols are still better (I am not sure, I was asking in my post, sorry for not being clear)

zinekeller 4y ago
Then you would generally try to look at probabilistic reliability. One way systems like digital broadcasting requires probabilistic reliability so a lot of research was done here. Digital broadcasting has produced a lot of answers in this area, as you need to deal with both data corruption and missing data. On the data corruption side, you would want to implement a forward-error correction (FEC, https://en.wikipedia.org/wiki/Error_correction_code#Forward_error_correction https://en.wikipedia.org/wiki/Error_correction_code#Forward_...) to allow recovery of corrupted packets, while on missing data it can be mitigated by fountain codes (https://en.wikipedia.org/wiki/Fountain_code https://en.wikipedia.org/wiki/Fountain_code) that is basically FEC but deals with missing pieces instead of incorrect data. This of course does not guarantee 100% delivery but in practical terms this is humanly reliable.
- _8j50 4y ago
  Thank you, that certainly gives me a lot to think about in this area.
simmervigor 4y ago
You could build this as an application mapping over QUIC. Within a single QUIC connection use reliable streams for the metadata delivery, and unreliable datagrams [1] for the payload. The benefit of a single connection is shared security context and shared fates of different datatype transfers. [1] - https://www.rfc-editor.org/rfc/rfc9221.html https://www.rfc-editor.org/rfc/rfc9221.html