3 ms·

I've been dying to write a protocol to replace both TCP and UDP (me being naive, I think I could write a decent one). I feel like there's fundamentally really n

I've been dying to write a protocol to replace both TCP and UDP (me being naive, I think I could write a decent one). I feel like there's fundamentally really no good reason to have multiple protocols, because fundamentally, neither is UDP 100% unreliable, nor is TCP 100% reliable. I think what we should really have is just one protocol with adjustable tolerance parameters. But I haven't had the time to put much thought into stuff like congestion control, so I don't know what it involves yet. I'm glad someone is doing something similar though.

riffraff 10y ago
UDP and TCP do not differ just because of reliability, they also differ because TCP is a stream protocol and UDP is not. By not being designed to handle streams, UDP can sidestep a ton of issues. So, the adjustable parameters in your protocol should be a lot, methinks.
- wfunction 10y ago
  I see why it appears that way, but I wasn't trying to suggest that reliability was the only distinction. However, I'm not sure I see such a fundamental distinction between the two that requires these to be two separate protocols. I can imagine parameters like "max wait time allowed before packet is skipped", "max contiguous packets allowed to be skipped", "whether to deliver packets that arrive out of order", stuff like that. Because ultimately TCP is at the extreme of retrying forever until every byte is received in order (or, well, until a timeout happens and we give up), and UDP is at the extreme of not retrying at all. It shouldn't be that bad to interpolate between the two I think?
  - busterarm 10y ago
    The tunability aspect of your idea scares me a little. When you choose TCP or UDP, you are making some guarantees about how the connection is going to behave. My fear here is that you lose those guarantees once you allow configuration of the connection using your one protocol. ...And while sure, you can have that negotiated when the connection is set up, there would be nothing from preventing one side of the connection from fudging/cheating/outright lying. I wouldn't put that past people at all -- especially networking hardware vendors. Any attempt you would implement to mitigate this just adds unnecessary data transmission overhead.
    - wfunction 10y ago
      > When you choose TCP or UDP, you are making some guarantees about how the connection is going to behave. Huh? With UDP, the thing missing is a guarantee about how the connection is going to behave (assuming you call it a "connection" in the first place). I'm just talking about interpolating between guarantees and no guarantees.
      - djrogers 10y ago
        > With UDP, the thing missing is a guarantee Not so - with UDP I know that the endpoint won't discard packets that arrive out of order, and won't bother me with retrans requests for stale packets that didn't arrive... different priorities, different kinds of 'guarantees'
  - nrjdhsbsid 10y ago
    You can configure retries in TCP. There's a ton of differences in how they're implemented underneath, merging them might be possible but you would probably end up with something that was worse for streams than TCP and worse for datagrams than UDP even if it could do both. There's three different retry strategies in TCP I know of. Like 7 or 8 popular congestion control algorithms, many types of queuing disciplines. Some fundamental differences like the SYN/ACK handshake, SYN cookies, state table conntrack, packet sequencing stuff. Tcp slow start and windows. Tons of optional features. The TCP stack of any newer OS is monstrous. A lot of kernel devs would cringe at the thought of adding more to that :)
    - wfunction 10y ago
      > You can configure retries in TCP. There's a ton of differences in how they're implemented underneath, merging them might be possible but you would probably end up with something that was worse for streams than TCP and worse for datagrams than UDP even if it could do both. I don't think that's the same kind of retry as the one I'm talking about. In TCP, no matter what the retry algorithm is, TCP has to deliver data in order. Whereas I'm talking about being able to tune how much you're willing to retry receiving the data until you give up and return it to the caller out of order. It's not about the algorithm so much as the resulting semantics.
- deleted 10y ago
  [deleted]
IshKebab 10y ago
SCTP allows configurable delivery parameters - you can choose whether data should be reliable, how many retransmissions, whether it needs to be in order, etc. It's used for WebRTC data channels: https://www.html5rocks.com/en/tutorials/webrtc/datachannels/ https://www.html5rocks.com/en/tutorials/webrtc/datachannels/ Also worth checking out enet: http://enet.bespin.org/Features.html http://enet.bespin.org/Features.html
- wfunction 10y ago
  Thanks, I'll take a look. From your description and my previous thoughts about the problem, the problem I see is that is "whether data should be reliable" and "how many transmissions" are not the sorts of parameters you want. Nobody cares about the number of retransmissions; people care about how much time is spent transmitting. And no connection is ever 100% reliable, so you want more than a Boolean.
hueving 10y ago
If you allow tcp to drop/re-order packets, the entire abstraction of a stream protocol is gone. The application now has to understand packet boundaries, which packets are missing, etc. So to me it sounds like you just want to write a UDP protocol with some extra reliability. Most applications built on top of UDP do this already, but maybe there would be some value in a general library for UDP protocols.
- wfunction 10y ago
  > If you allow tcp to drop/re-order packets, the entire abstraction of a stream protocol is gone. The application now has to understand packet boundaries, which packets are missing, etc. I don't think so. On the receiver's side, the application doesn't need to know packets at all -- it just deals with a stream of bytes where some portions may be dropped or come out of order. The packet nature would be abstracted away. On the sender's side, packets only become relevant if the application wants to get optimal behavior with respect to the packet size (in which case it would be already doing its own manual UDP). If it just wants "good enough" generic behavior (which is what TCP has always been providing for everyone) then it can again ignore the packet nature and just treat the connection as a stream where some portions may be dropped and/or reordered. Think about how useful this could be in so many common situations: when you're trying to download some file, for example, you don't need TCP -- because it doesn't matter if the data comes out of order. You just need to make sure you get everything eventually and that the data isn't corrupted. You don't want UDP either, which would require you to implement your own feedback/congestion control/etc. mechanisms from scratch. There's a nice sweet spot in the middle waiting to be hit. And it baffles me that it still doesn't seem to have been done.
  - DasIch 10y ago
    The receiver now has to figure out what is missing from a stream of data of which anything could be missing. That's equivalent to dealing with missing packets only on a byte level. If anything, that's harder and less efficient to deal with.

riffraff 10y ago

UDP and TCP do not differ just because of reliability, they also differ because TCP is a stream protocol and UDP is not. By not being designed to handle streams, UDP can sidestep a ton of issues. So, the adjustable parameters in your protocol should be a lot, methinks.

wfunction 10y ago
I see why it appears that way, but I wasn't trying to suggest that reliability was the only distinction. However, I'm not sure I see such a fundamental distinction between the two that requires these to be two separate protocols. I can imagine parameters like "max wait time allowed before packet is skipped", "max contiguous packets allowed to be skipped", "whether to deliver packets that arrive out of order", stuff like that. Because ultimately TCP is at the extreme of retrying forever until every byte is received in order (or, well, until a timeout happens and we give up), and UDP is at the extreme of not retrying at all. It shouldn't be that bad to interpolate between the two I think?
- busterarm 10y ago
  The tunability aspect of your idea scares me a little. When you choose TCP or UDP, you are making some guarantees about how the connection is going to behave. My fear here is that you lose those guarantees once you allow configuration of the connection using your one protocol. ...And while sure, you can have that negotiated when the connection is set up, there would be nothing from preventing one side of the connection from fudging/cheating/outright lying. I wouldn't put that past people at all -- especially networking hardware vendors. Any attempt you would implement to mitigate this just adds unnecessary data transmission overhead.
  - wfunction 10y ago
    > When you choose TCP or UDP, you are making some guarantees about how the connection is going to behave. Huh? With UDP, the thing missing is a guarantee about how the connection is going to behave (assuming you call it a "connection" in the first place). I'm just talking about interpolating between guarantees and no guarantees.
    - djrogers 10y ago
      > With UDP, the thing missing is a guarantee Not so - with UDP I know that the endpoint won't discard packets that arrive out of order, and won't bother me with retrans requests for stale packets that didn't arrive... different priorities, different kinds of 'guarantees'
- nrjdhsbsid 10y ago
  You can configure retries in TCP. There's a ton of differences in how they're implemented underneath, merging them might be possible but you would probably end up with something that was worse for streams than TCP and worse for datagrams than UDP even if it could do both. There's three different retry strategies in TCP I know of. Like 7 or 8 popular congestion control algorithms, many types of queuing disciplines. Some fundamental differences like the SYN/ACK handshake, SYN cookies, state table conntrack, packet sequencing stuff. Tcp slow start and windows. Tons of optional features. The TCP stack of any newer OS is monstrous. A lot of kernel devs would cringe at the thought of adding more to that :)
  - wfunction 10y ago
    > You can configure retries in TCP. There's a ton of differences in how they're implemented underneath, merging them might be possible but you would probably end up with something that was worse for streams than TCP and worse for datagrams than UDP even if it could do both. I don't think that's the same kind of retry as the one I'm talking about. In TCP, no matter what the retry algorithm is, TCP has to deliver data in order. Whereas I'm talking about being able to tune how much you're willing to retry receiving the data until you give up and return it to the caller out of order. It's not about the algorithm so much as the resulting semantics.
deleted 10y ago
[deleted]

IshKebab 10y ago

SCTP allows configurable delivery parameters - you can choose whether data should be reliable, how many retransmissions, whether it needs to be in order, etc. It's used for WebRTC data channels: https://www.html5rocks.com/en/tutorials/webrtc/datachannels/ https://www.html5rocks.com/en/tutorials/webrtc/datachannels/ Also worth checking out enet: http://enet.bespin.org/Features.html http://enet.bespin.org/Features.html

wfunction 10y ago
Thanks, I'll take a look. From your description and my previous thoughts about the problem, the problem I see is that is "whether data should be reliable" and "how many transmissions" are not the sorts of parameters you want. Nobody cares about the number of retransmissions; people care about how much time is spent transmitting. And no connection is ever 100% reliable, so you want more than a Boolean.

hueving 10y ago

If you allow tcp to drop/re-order packets, the entire abstraction of a stream protocol is gone. The application now has to understand packet boundaries, which packets are missing, etc. So to me it sounds like you just want to write a UDP protocol with some extra reliability. Most applications built on top of UDP do this already, but maybe there would be some value in a general library for UDP protocols.

wfunction 10y ago
> If you allow tcp to drop/re-order packets, the entire abstraction of a stream protocol is gone. The application now has to understand packet boundaries, which packets are missing, etc. I don't think so. On the receiver's side, the application doesn't need to know packets at all -- it just deals with a stream of bytes where some portions may be dropped or come out of order. The packet nature would be abstracted away. On the sender's side, packets only become relevant if the application wants to get optimal behavior with respect to the packet size (in which case it would be already doing its own manual UDP). If it just wants "good enough" generic behavior (which is what TCP has always been providing for everyone) then it can again ignore the packet nature and just treat the connection as a stream where some portions may be dropped and/or reordered. Think about how useful this could be in so many common situations: when you're trying to download some file, for example, you don't need TCP -- because it doesn't matter if the data comes out of order. You just need to make sure you get everything eventually and that the data isn't corrupted. You don't want UDP either, which would require you to implement your own feedback/congestion control/etc. mechanisms from scratch. There's a nice sweet spot in the middle waiting to be hit. And it baffles me that it still doesn't seem to have been done.
- DasIch 10y ago
  The receiver now has to figure out what is missing from a stream of data of which anything could be missing. That's equivalent to dealing with missing packets only on a byte level. If anything, that's harder and less efficient to deal with.