7 ms·

I'm starting to think that TCP is past its prime and UDP based protocols (like QUIC) are the way forward. The problem is exacerbated by TLS encryption. It requ

by exDM69 3y ago

I'm starting to think that TCP is past its prime and UDP based protocols (like QUIC) are the way forward. The problem is exacerbated by TLS encryption. It requires six packets, or three round trip times, to establish a TLS over TCP connection. That is hundreds of milliseconds before a single byte of payload can be transferred. The QUIC protocol combines the three way handshake and the three way key exchange to half the time I takes to set up a connection and provides a fast start (0rtt) for resuming a previously configured connection. Additionally TCP connections do not survive a changed IP address like when changing from wifi to mobile data. The stream model of TCP with head of the line blocking is conceptually easy but ultimately the wrong model for transmitting blobs of data. Rejecting a valid packet because the order got shuffled is wasteful, when we could just as easily reassemble the packets on the receiver side (even outside the sliding window). There are a lot of times when the stream model is a good choice (dynamically generated data) but I would argue that sending sized blobs (like images) is a larger volume of traffic in terms of bytes transferred. But the OP is right, writing custom UDP protocol is fraught with peril. DDoS amplification is just one way to screw up.

defrost 3y ago
TCP is overkill for the vast bulk of modern data transmission by volume; "short hops" within and between data centres on backbones that have near zero error. TCP was designed for higher error rates on less reliable paths; from the clustered data centres out to the more remote home computers today is likely better than 1980s internet. There are good proposals for low error high bandwidth TCP-lite replacements that are gaining ground.
- agumonkey 3y ago
  Could you name a few of these proposals ?
  - defrost 3y ago
    Yes. Homa for one: * (2018) https://people.csail.mit.edu/alizadeh/papers/homa-sigcomm18.pdf https://people.csail.mit.edu/alizadeh/papers/homa-sigcomm18.... * (2022) https://arxiv.org/abs/2210.00714 https://arxiv.org/abs/2210.00714 * (2022) https://www.theregister.com/2022/07/27/replace_tcp_datacenter/ https://www.theregister.com/2022/07/27/replace_tcp_datacente... * https://github.com/PlatformLab/Homa https://github.com/PlatformLab/Homa
    - agumonkey 3y ago
      Thanks :)
- goeiedaggoeie 3y ago
  Error rates are still high. You quickly find that if you have latency sensitive information that you need to ship over the public internet, like live video.
  - midasuni 3y ago
    Depends what you mean by high, and where you’re going to.
  - dmurray 3y ago
    Also, you can trade off bandwidth for error rate at a lower level of the networking stack, so if you know your applications will use error- correcting protocols like TCP, you can make your switches and routers talk to each other a little faster.
- cduzz 3y ago
  I'm not sure I'd want to switch to a pure UDP world just because there's lots of bandwidth in the data center. Any situation where Moore's Law works against you will run into problems unless there's back-pressure built into the protocol -- IE whenever you've got 100 clients talking to one aggregation point you'll get kicked in the teeth, regardless of if the endpoints are 100mb with the aggregation point being bonded 1gb links or the common media is 100gb and the aggregation point is ECMPed 400gb -- there are more of them than there is of you... And -- building back-pressure into protocols is not trivial.... Getting it "for free" with TCP has been a net gain.
- jeltz 3y ago
  TCP works terribly on connections with a high error rate. This is one of the main reasons for wanting to replace it with something like QUIC.
touisteur 3y ago
For multihoming, MPTCP is an interesting alternative and it's full of interesting features. Once it's more spread out it'd be interesting to know which features are really used. I haven't seen SCTP mentionned in here so I'll just say that learning and using it, first in trying to find 'a better UDP' and then in anger to solve a multihoming/high-availability problem (which ended up in TCP + checkpoint/restore anyway...) was an eye opening experience on 'what could have been', especially having it work over UDP. Now QUIC is there, gathering a big chunk of the SCTP featureset and adding proper encryption support (where IIRC SCTP only had TLS per stream).
- EvanAnderson 3y ago
  WebRTC uses SCTP over UDP, so there are Free implementations out there you can grab if you want to give it a try. It's a shame that the Internet has ossified to the point that we'll never have any protocols riding directly on top of IP besides TCP, UDP, and ICMP, but tunneling SCTP over UDP doesn't add ridiculous overhead.
  - oneshtein 3y ago
    [flagged]
    - IX-103 3y ago
      Lol. Good one. These protocols are completely independent of the content that you can send over them. For the most part they reuse TLS so the encryption is the same (and uses the same certificate authorities) so any man-in-the-middle proxies, like those used by corporations, are technically able to intercept them to monitor content just like they can for https. (Though personally those sorts of things make me feel ill ) Good luck banning these protocols as well. They're perfectly valid IP packets so identifying them is going to be hard.
    - Kwpolska 3y ago
      Protocols that aren't HTTPS (like SMTP) are still alive. The reason for no new protocols is that some network infrastructure will consider non-TCP/UDP/ICMP packets badly formed and reject them.
goeiedaggoeie 3y ago
protocols like quic and srt (used for video) are great, forward error/erasure correction is something I would also mention as a large part of the rise of UDP over TCP based transfer/protocols.
Olreich 3y ago
I don't think skipping the handshake is much of a benefit to QUIC. TCP was designed to have long-lived connections. Mobile networks are moving toward keeping your external IP consistent between handoffs, handling the routing internally. Fixed networks will connect to servers once every couple of minutes at most. QUIC mostly affects time-to-first-paint, which is mostly of interest for the slowest connections, which is nice, but becoming less and less relevant over time. It's not like speeding up the connectivity time is going to matter when all gains in connectivity time will be used for additional advertising and tracking against the end-user. The main value propositions of QUIC for me are in the encryption earlier in the connection and in the reduced infrastructure usage on the server side.
- withinboredom 3y ago
  UDP is connectionless, so that aspect isn't even comparable to TCP.
  - tonyarkles 3y ago
    I mean, yes, it is but many (not all) protocols built on top of it are generally still “connection-oriented”. QUIC, for example, still results in a “connection” it’s just implemented at the application layer instead of as part of the UDP protocol itself.
    - withinboredom 3y ago
      That still isn't even comparable. The routers along the path of a UDP packet may not even be tracking the flow (unlike in TCP), and keeping it up. A "connection" in TCP is a very big beast, with routers and computers tracking them (conntrack and friends) to keep them alive (or not) and ensure everything runs smoothly. There is none of that with UDP. So yeah, there might be an application-level concept of a connection, but that isn't guaranteed by a single bit of infrastructure on the internet.
      - oasisbob 3y ago
superkuh 3y ago
>The QUIC protocol combines the three way handshake and the three way key exchange to half the time I takes to set up a connection and provides a fast start (0rtt) for resuming a previously configured connection. UDP is okay and has some problems. QUIC is not the universal solution for them. The lifetime of that QUIC setup will only be ~90 days. Because with every QUIC implementation in existence use of self-signed certs is not enabled by default and TLS-less connections are not allowed in the standard. This hold especially true for QUIC based HTTP/3 libs. So as soon as something in the TLS breaks or drops you your QUIC based service is inaccessible to 99% of people on Earth. That's fine, good even, for Microsoft or Google. But that kind of built in fragility, transience, and corporate centralization/dependence is really bad for hosting a personal website or service. QUIC is for corporate persons usage and it's design reflects that. It's terrible for human persons and makes no concession for human person use cases.
- pixl97 3y ago
  Because human person traffic is a tiny percentage of actual internet traffic. Where is most internet traffic? To places like Netflix or other streaming services. Then you have the top X sites that serve most of the rest of the traffic, a significant percentage of that to cell phones and all of it needing encrypted. You'll just have to accept no realistically secure protocol will be universal. This last tiny wedge of traffic is either going to disappear in the noise of bots in automated attacks, or we'll have to find some reasonably secure way of letting it co-exist.
- bmicraft 3y ago
  It's really not that hard to setup a domain with letsencrypt. If you use dns verification it's even almost trivially easy to use it for internal hosts as well.
  - superkuh 3y ago
    It is till it isn't. When ACME2 is sunset and ACME3 comes along. The next time LE's root cert changes. The next time any part of the incredibly complexity hidden by and within acme clients breaks because of an update on your system. The time you host your abortion clinic site and get sued in Texas. Or your scientific journal article mirror gets attacked by big publishers, they won't go after you when they can just put pressure on LE to drop you and you become unvistable. Putting all your chips in one basket, even such a benevolent and great one as LE, is a bad idea. Even dot .org was almost lost to private equity and went nasty. LE as it becomes more important becomes a juicier target for all forms of pressure and less useful to human persons.
the8472 3y ago
> Rejecting a valid packet because the order got shuffled is wasteful, This is not what happens. The data isn't dropped as long as there are only a few reordered segments. And with SACK the receiver can also acknowledge them. It's just not provided to the application layer because something in-between is still missing. This adds some latency if you're multiplexing multiple things over a single stream but it's not a loss in throughput. > The stream model of TCP with head of the line blocking is conceptually easy but ultimately the wrong model for transmitting blobs of data. A problem of google's own creation. > The problem is exacerbated by TLS encryption. It requires six packets, or three round trip times, to establish a TLS over TCP connection. That is hundreds of milliseconds before a single byte of payload can be transferred. Arguably the problem got exacerbated by the NSA and user-hostile ISPs. If they didn't push the entire internet into a low-trust equilibrium we could stick to simple protocols instead of having to expend more resources and increase complexity to ultimately get to the same state of not-being-intercepted as we had been before. A better response to a red queen's race is to dethrone the queen, not to build better running shoes. But yes, yes... these sure are nice shoes.
jcul 3y ago
TLS does have session IDs / session tickets, which I believe can speed up session resumption. There is also TCP fast open, but I don't think it is used in practice. I don't think I've ever noticed it used when looking at a TCP capture. Though I'm sure either TLS session IDs or tickets are. I can't remember if one is more common that the other, it's been a while since I worked on that stuff directly.
lamontcg 3y ago
> UDP based protocols (like QUIC) are the way forward. Sure, but QUIC was written by experts who probably had been burned in the past by thinking they could just throw away TCP and profit. Things like slow start and congestion control are there for really good reasons and you can't just casually toss them away like anachronisms. Modern networks are really big, but ultimately things like switch buffers are still finite. Rolling your own UDP based protocol is kind of like rolling your own cryptography. Amazon in 2001 was using lots of UDP on top of Tibco Rendezvous when I got there and it absolutely melted the fuck down in Christmas of 2003. All of the FAANG-class companies have probably rediscovered this kind of lesson independently, maybe multiple times. I certainly remember chuckling at some Facebook press release in ~2007 when they were bragging about switching to UDP and wonder how that turned out.
deleted 3y ago
[deleted]

by exDM69 3y ago

defrost 3y ago

TCP is overkill for the vast bulk of modern data transmission by volume; "short hops" within and between data centres on backbones that have near zero error. TCP was designed for higher error rates on less reliable paths; from the clustered data centres out to the more remote home computers today is likely better than 1980s internet. There are good proposals for low error high bandwidth TCP-lite replacements that are gaining ground.

agumonkey 3y ago
Could you name a few of these proposals ?
- defrost 3y ago
  Yes. Homa for one: * (2018) https://people.csail.mit.edu/alizadeh/papers/homa-sigcomm18.pdf https://people.csail.mit.edu/alizadeh/papers/homa-sigcomm18.... * (2022) https://arxiv.org/abs/2210.00714 https://arxiv.org/abs/2210.00714 * (2022) https://www.theregister.com/2022/07/27/replace_tcp_datacenter/ https://www.theregister.com/2022/07/27/replace_tcp_datacente... * https://github.com/PlatformLab/Homa https://github.com/PlatformLab/Homa
  - agumonkey 3y ago
    Thanks :)
goeiedaggoeie 3y ago
Error rates are still high. You quickly find that if you have latency sensitive information that you need to ship over the public internet, like live video.
- midasuni 3y ago
  Depends what you mean by high, and where you’re going to.
- dmurray 3y ago
  Also, you can trade off bandwidth for error rate at a lower level of the networking stack, so if you know your applications will use error- correcting protocols like TCP, you can make your switches and routers talk to each other a little faster.
cduzz 3y ago
I'm not sure I'd want to switch to a pure UDP world just because there's lots of bandwidth in the data center. Any situation where Moore's Law works against you will run into problems unless there's back-pressure built into the protocol -- IE whenever you've got 100 clients talking to one aggregation point you'll get kicked in the teeth, regardless of if the endpoints are 100mb with the aggregation point being bonded 1gb links or the common media is 100gb and the aggregation point is ECMPed 400gb -- there are more of them than there is of you... And -- building back-pressure into protocols is not trivial.... Getting it "for free" with TCP has been a net gain.
jeltz 3y ago
TCP works terribly on connections with a high error rate. This is one of the main reasons for wanting to replace it with something like QUIC.

touisteur 3y ago

For multihoming, MPTCP is an interesting alternative and it's full of interesting features. Once it's more spread out it'd be interesting to know which features are really used. I haven't seen SCTP mentionned in here so I'll just say that learning and using it, first in trying to find 'a better UDP' and then in anger to solve a multihoming/high-availability problem (which ended up in TCP + checkpoint/restore anyway...) was an eye opening experience on 'what could have been', especially having it work over UDP. Now QUIC is there, gathering a big chunk of the SCTP featureset and adding proper encryption support (where IIRC SCTP only had TLS per stream).

EvanAnderson 3y ago
WebRTC uses SCTP over UDP, so there are Free implementations out there you can grab if you want to give it a try. It's a shame that the Internet has ossified to the point that we'll never have any protocols riding directly on top of IP besides TCP, UDP, and ICMP, but tunneling SCTP over UDP doesn't add ridiculous overhead.
- oneshtein 3y ago
  [flagged]
  - IX-103 3y ago
    Lol. Good one. These protocols are completely independent of the content that you can send over them. For the most part they reuse TLS so the encryption is the same (and uses the same certificate authorities) so any man-in-the-middle proxies, like those used by corporations, are technically able to intercept them to monitor content just like they can for https. (Though personally those sorts of things make me feel ill ) Good luck banning these protocols as well. They're perfectly valid IP packets so identifying them is going to be hard.
  - Kwpolska 3y ago
    Protocols that aren't HTTPS (like SMTP) are still alive. The reason for no new protocols is that some network infrastructure will consider non-TCP/UDP/ICMP packets badly formed and reject them.

goeiedaggoeie 3y ago

protocols like quic and srt (used for video) are great, forward error/erasure correction is something I would also mention as a large part of the rise of UDP over TCP based transfer/protocols.

Olreich 3y ago

I don't think skipping the handshake is much of a benefit to QUIC. TCP was designed to have long-lived connections. Mobile networks are moving toward keeping your external IP consistent between handoffs, handling the routing internally. Fixed networks will connect to servers once every couple of minutes at most. QUIC mostly affects time-to-first-paint, which is mostly of interest for the slowest connections, which is nice, but becoming less and less relevant over time. It's not like speeding up the connectivity time is going to matter when all gains in connectivity time will be used for additional advertising and tracking against the end-user. The main value propositions of QUIC for me are in the encryption earlier in the connection and in the reduced infrastructure usage on the server side.

withinboredom 3y ago
UDP is connectionless, so that aspect isn't even comparable to TCP.
- tonyarkles 3y ago
  I mean, yes, it is but many (not all) protocols built on top of it are generally still “connection-oriented”. QUIC, for example, still results in a “connection” it’s just implemented at the application layer instead of as part of the UDP protocol itself.
  - withinboredom 3y ago
    That still isn't even comparable. The routers along the path of a UDP packet may not even be tracking the flow (unlike in TCP), and keeping it up. A "connection" in TCP is a very big beast, with routers and computers tracking them (conntrack and friends) to keep them alive (or not) and ensure everything runs smoothly. There is none of that with UDP. So yeah, there might be an application-level concept of a connection, but that isn't guaranteed by a single bit of infrastructure on the internet.
    - oasisbob 3y ago

superkuh 3y ago

>The QUIC protocol combines the three way handshake and the three way key exchange to half the time I takes to set up a connection and provides a fast start (0rtt) for resuming a previously configured connection. UDP is okay and has some problems. QUIC is not the universal solution for them. The lifetime of that QUIC setup will only be ~90 days. Because with every QUIC implementation in existence use of self-signed certs is not enabled by default and TLS-less connections are not allowed in the standard. This hold especially true for QUIC based HTTP/3 libs. So as soon as something in the TLS breaks or drops you your QUIC based service is inaccessible to 99% of people on Earth. That's fine, good even, for Microsoft or Google. But that kind of built in fragility, transience, and corporate centralization/dependence is really bad for hosting a personal website or service. QUIC is for corporate persons usage and it's design reflects that. It's terrible for human persons and makes no concession for human person use cases.

pixl97 3y ago
Because human person traffic is a tiny percentage of actual internet traffic. Where is most internet traffic? To places like Netflix or other streaming services. Then you have the top X sites that serve most of the rest of the traffic, a significant percentage of that to cell phones and all of it needing encrypted. You'll just have to accept no realistically secure protocol will be universal. This last tiny wedge of traffic is either going to disappear in the noise of bots in automated attacks, or we'll have to find some reasonably secure way of letting it co-exist.
bmicraft 3y ago
It's really not that hard to setup a domain with letsencrypt. If you use dns verification it's even almost trivially easy to use it for internal hosts as well.
- superkuh 3y ago
  It is till it isn't. When ACME2 is sunset and ACME3 comes along. The next time LE's root cert changes. The next time any part of the incredibly complexity hidden by and within acme clients breaks because of an update on your system. The time you host your abortion clinic site and get sued in Texas. Or your scientific journal article mirror gets attacked by big publishers, they won't go after you when they can just put pressure on LE to drop you and you become unvistable. Putting all your chips in one basket, even such a benevolent and great one as LE, is a bad idea. Even dot .org was almost lost to private equity and went nasty. LE as it becomes more important becomes a juicier target for all forms of pressure and less useful to human persons.

the8472 3y ago

> Rejecting a valid packet because the order got shuffled is wasteful, This is not what happens. The data isn't dropped as long as there are only a few reordered segments. And with SACK the receiver can also acknowledge them. It's just not provided to the application layer because something in-between is still missing. This adds some latency if you're multiplexing multiple things over a single stream but it's not a loss in throughput. > The stream model of TCP with head of the line blocking is conceptually easy but ultimately the wrong model for transmitting blobs of data. A problem of google's own creation. > The problem is exacerbated by TLS encryption. It requires six packets, or three round trip times, to establish a TLS over TCP connection. That is hundreds of milliseconds before a single byte of payload can be transferred. Arguably the problem got exacerbated by the NSA and user-hostile ISPs. If they didn't push the entire internet into a low-trust equilibrium we could stick to simple protocols instead of having to expend more resources and increase complexity to ultimately get to the same state of not-being-intercepted as we had been before. A better response to a red queen's race is to dethrone the queen, not to build better running shoes. But yes, yes... these sure are nice shoes.

jcul 3y ago

TLS does have session IDs / session tickets, which I believe can speed up session resumption. There is also TCP fast open, but I don't think it is used in practice. I don't think I've ever noticed it used when looking at a TCP capture. Though I'm sure either TLS session IDs or tickets are. I can't remember if one is more common that the other, it's been a while since I worked on that stuff directly.

lamontcg 3y ago

> UDP based protocols (like QUIC) are the way forward. Sure, but QUIC was written by experts who probably had been burned in the past by thinking they could just throw away TCP and profit. Things like slow start and congestion control are there for really good reasons and you can't just casually toss them away like anachronisms. Modern networks are really big, but ultimately things like switch buffers are still finite. Rolling your own UDP based protocol is kind of like rolling your own cryptography. Amazon in 2001 was using lots of UDP on top of Tibco Rendezvous when I got there and it absolutely melted the fuck down in Christmas of 2003. All of the FAANG-class companies have probably rediscovered this kind of lesson independently, maybe multiple times. I certainly remember chuckling at some Facebook press release in ~2007 when they were bragging about switching to UDP and wonder how that turned out.

deleted 3y ago

[deleted]