6 ms·

QUIC and the end of TCP sockets

62 points by charles_irl 1y ago

nickysielicki 1y ago
> Running in user space offers more flexibility for resource management and experimentation. I stopped reading here. This isn’t really an essential property of QUIC, there’s a lot of good reasons to eventually try to implement this in the kernel. https://lwn.net/Articles/1029851/ https://lwn.net/Articles/1029851/
- lxgr 1y ago
  Maybe not an essential property of QUIC, but definitely one of not using TCP. Most OSes don't let you send raw TCP segments without superuser privileges, so you can't just bring your own TCP congestion control algorithm in the userspace, unless you also wrap your custom TCP segments in UDP.
  - bayindirh 1y ago
    Why do the hard work if the same thing can be done by the kernel, or even by the card itself? Drown other applications for your own benefit?
    - lxgr 1y ago
      > Why do the hard work if the same thing can be done by the kernel, or even by the card itself? How would you swap out the TCP congestion control algorithm in an OS, or even hardware, you don't control? > Drown other applications for your own benefit? Fairness equivalent to classic TCP is a design goal of practically all alternative algorithms, so I'm not sure what you're implying. It's entirely possible to improve responsivity without compromising on fairness, as e.g. BBR has shown.
      - neilalexander 1y ago
        > How would you swap out the TCP congestion control algorithm in an OS, or even hardware, you don't control? On the contrary, introducing their own novel/mysterious/poorly implemented congestion control algorithms is not a thing I want userspace applications doing.
        lxgr 1y ago
        Fortunately you don't get any say in what my userspace applications do on my own hardware. And if you worry about hostile applications on your own hardware, the OS is an excellent point to limit what they can do – including overwhelming your network interface.
        neilalexander 1y ago
        > the OS is an excellent point to limit what they can do – including overwhelming your network interface One might even call this "congestion control"!
        lxgr 1y ago
        No, congestion control in the network sense is about congestion at choke points in packet networks due to higher inflow than outflow rates. While you're still on your own host, your OS has complete visibility into which applications are currently intending to send data and can schedule them much more directly.
    - jeffbee 1y ago
      "Drown other applications" is unfortunately exactly what happens when you let the Linux kernel run your TCP stack. Profile your application and you may discover that your CPUs are being spent running the protocol stack on behalf of other applications.
      - lxgr 1y ago
        What do you mean by "other applications"?
        jeffbee 1y ago
        I mean when your application sends on a socket, the kernel may also send and receive traffic for another task while it's in the syscall, just for funsies, and this is true even if your applications are containerized and you believe their CPU cores are dedicated to the container.
        lxgr 1y ago
        Ah, but that's an OS implementation problem, not one with TCP or QUIC, no?
        jeffbee 1y ago
        Sure, but the ready appeal of QUIC is that it is in user space by nature, while Linux ties TCP to the kernel. You either need special privileges to run user space TCP on Linux, or you need a different operating system kernel altogether.
lxgr 1y ago
> QUIC’s design intentionally separates the wire protocol from the congestion control algorithm Is that not the case for TCP as well? Most congestion control algorithms just assign new meanings to existing wire-level flags (e.g. duplicate ACKs), or even only change sender-side behavior. > QUIC gives* control back to application developers to tailor congestion control to their use case* That's what it actually does: It moves the congestion control implementation from the OS to user space. In that sense, it's the same tradeoff as containers vs. regular binaries linking to shared libraries: Great if your applications are updated more often than your OS; not so great if it's the other way around.
- bayindirh 1y ago
  > QUIC gives control back to application developers to tailor congestion control to their use case If I understood modern application development correctly, this interprets as "The developers will import another library which they don't understand and will wreak havoc on other applications' data streams by only optimizing stuff for themselves". Again, if I remember correctly, an OS is "the layer which manages the sharing of limited resources among many processes which requests/needs it", and the OS can do system-wide, per socket congestion control without any effort because of the vantage point it has over networking layer. Assuming that every application will do congestion control correctly while not choking everyone else even unintentionally with user space's limited visibility is absurd at worst, and wishful thinking at best. The whole ordeal is direct violation with application separation coming with protected mode.
  - lxgr 1y ago
    > Assuming that every application will do congestion control correctly while not choking everyone else even unintentionally with user space's limited visibility is absurd at worst, and wishful thinking at best. Why? All practically used variants of TCP achieve fairness at the bottleneck without any central arbiter or explicit view of the upstream congestion situation. Besides, UDP has never had congestion control and has been around for decades. > The whole ordeal is direct violation with application separation coming with protected mode. [...] The whole ordeal is direct violation with application separation coming with protected mode. Are you concerned about fairness between applications on the same OS or fairness at the bottleneck, usually upstream from both application and OS/host? In the former case, the OS always gets the last word, whether you're using TCP or your own homebrew congestion control via UDP, and you can make sure that each application gets the exact same fair egress rate if required. In the latter case, nothing prevents anyone from patching their kernel and running "unfair TCP" today.
IgorPartola 1y ago
The obnoxious thing is that overly aggressive firewalls have killed any IP protocols that are not TCP or UDP. Even ICMP is often blocked or partially blocked. In the mean time we could have had nice things: https://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr... SCTP would be a fantastic protocol for HTTP/HTTPS. Pipelining, multi homing, multi streaming, oh my.
- Hydraulix989 1y ago
  Not sure I follow. QUIC is just UDP with an extra header in the opaque payload. To a firewall, it looks just like UDP.
  - jabiko 1y ago
    That's the point he is making. QUIC has to be based on UDP because the networking stack is ossified enough to not allow the addition of any new Layer 4 protocol. It's not a huge drawback though.
    - Hydraulix989 1y ago
      It's an eight byte penalty for things needed for QUIC anyways.
  - p_l 1y ago
    It exists partially because of ossification of protocols that killed SCTP. As in, we wouldn't have to implement it on top of UDP
  - sedawkgrep 1y ago
    I think they’re saying that due to how firewalls are deployed, everything end up either being built on tcp or udp, instead of using existing (or building new) layer four protocols more suited to solving the problem like sctp, et al. I’m not sure I agree though, because many firewalls already pass other protocols today, like GRE, IPSEC, etc.
jklowden 1y ago
Note well: the claims about TCP come with some evidence, in the form of a graph. The claims for QUIC do not. Many of the claims are dubious. TCP has "no notion of multiple steams"? What are two sockets, then? What is poll(2)? The onus is on QUIC to explain why it’s better for the application to multiplex the socket than for the kernel to multiplex the device. AFAICT that question is assumed away in a deluge of words. If the author thinks it’s the "end of TCP sockets", show us the research, the published papers and meticulous detail. Then tell me again why I should eschew the services of TCP and absorb its complexity into my application.
- garganzol 1y ago
  Manipulative tone of the article title says it all. "The end of".
  - ashtakeaway 1y ago
    Really glad NextDNS blocked my click.
- tuetuopay 1y ago
  Even the TCP graph is dubious. Cubic being systematically above the link capacity makes me chuckle. Yes bufferbloat can have cubic "hug" a somewhat higher limit, but it still needs to start under the link capacity.
  - lxgr 1y ago
    That's easily explained, in the parts of the x axis missing in the plot it actually goes negative and pays back the borrowed bytes.
- cbluth 1y ago
  The obvious comparisons are between udp and tcp, and then between quic and ssh, given the notion of "multiple streams"
superkuh 1y ago
QUIC would be the end of the free internet if it ever "took over" but luckily it won't. It's not built to do so, it's only built for corporate use cases. QUIC implementations do not allow for anyone to connect to anyone else. Instead, because it was built entirely with corporate for-profit uses cases in mind and open-washed through the IETF, the idea of a third party coporation having to authenticate the identity of all connections is baked in. And 99.999% of QUIC libs, and the way they're shipped in clients, cannot even connect to a server without a third party corp first saying they know the end point and allow it. Fine for corporate/profit use cases where security of the monetary transactions is all that matters. Very much less fine for human uses cases where it forces centralization and easy control by our rapidly enshittifying authoritarian governments. QUIC is the antithesis to the concept of the internet and it's robustness and routing around damage.
- sleepydog 1y ago
  I guess you are referring to the TLS requirement? I guess I could see how on a more restrictive platform like a phone you could conceivably be prevented from accepting alternate CAs or self signed certificates.
- jauntywundrkind 1y ago
  There's a fairly far a long draft for replacing webrtc's SCTP with QUIC for doing p2p work. It doesn't seem to have any of these challenges, seems to be perfectly viable there for connecting peers. https://github.com/w3c/p2p-webtransport https://github.com/w3c/p2p-webtransport Alas alas, basically stalled out, afaik no implementation. I wish Microsoft (the spec author) or someone would pick this back up.
  - lxgr 1y ago
    WebRTC wraps SCTP in DTLS, so the "great challenge of encryption" has never been a problem there. It just uses self-signed certificates, which is maybe conceptually slightly clunky compared to "pure" anonymous TOFU, but allows reusing existing stacks.
- lxgr 1y ago
KaiserPro 1y ago
One thing to note is that using HTTP2.0 for anything other than "this is not how to design high throughput protocols" is unfair. At the time HTTP2.0's multiplexing was known to be bad for anything other than perfect, low latency networks. I hope this was because people had faith in better connectivity, rather than ignorance of how mobile and non-lan traffic worked. You should probably at least try QUIC now, but you can get past HOL blocking by having multiple TCP streams. Its super cheap, cheaper than QUIC.
- lxgr 1y ago
  > you can get past HOL blocking by having multiple TCP streams. Its super cheap, cheaper than QUIC And also super inefficient, since it duplicates the TLS handshake across streams and uses more resources in the OS and middleboxes (like them or hate them, they're a thing that might throttle you if you go too crazy with connection-level parallelism). That's on top of very poor fairness at bottlenecks (which is per TCP stream unless there's separate traffic policing).
  - KaiserPro 1y ago
    > since it duplicates the TLS handshake across streams Depends on your usecase of course. if its something thats only around for a minute, then yeah, but over an hour, makes no difference at all. QUIC eats more CPU per MB of data than plain TCP/TLS. but that might be solved by more modern offload/kernel optimisations
- tuetuopay 1y ago
  Even on perfect networks HoL blocking is an issue. If the receiving end of a set of stream blocks on one stream, it ends up blocking the whole connection. One stream stops due to backpressure, all streams stop.
  - KaiserPro 1y ago
    Oh yeah, but thats why HTTP2.0 was flawed from the start.
exabrial 1y ago
The most obnoxious thing about QUIC is I don't need encryption all of the time, actually the majority of the time. Useless overhead.
- lxgr 1y ago
  Even if you have nothing to hide and don't care about accidental or intentional data modification, the benefit of largely cutting out "clever" middleboxes alone is almost always worth it.
Ingon 1y ago
I used QUIC extensively to implement https://github.com/connet-dev/connet https://github.com/connet-dev/connet and while I'm super happy with how it turned out, I think QUIC currently suffers from some immaturity - most implementations are still ongoing/not production ready (for example in java) and in many cases it is only viewed as a way to power on HTTP/3, instead of being self-standing protocol/API that ppl can use (for example, trying to use quic in android). In any case, I'm optimistic that QUIC has a bright future. I don't expect it to replace TCP, but give us another tool we can use when it is called for.
mannyv 1y ago
Leaving congestion control to apps means that congestion control at a system level will be impossible. Is that really a QUIC thing?
yencabulator 1y ago
The article starts with > Why QUIC’s user-space transport lets us ‘kill’ the old app-level event loop But then doesn't seem to mention that topic ever again. I don't see how QUIC changes that much.

62 points by charles_irl 1y ago

> Running in user space offers more flexibility for resource management and experimentation. I stopped reading here. This isn’t really an essential property of QUIC, there’s a lot of good reasons to eventually try to implement this in the kernel. https://lwn.net/Articles/1029851/ https://lwn.net/Articles/1029851/

lxgr 1y ago
Maybe not an essential property of QUIC, but definitely one of not using TCP. Most OSes don't let you send raw TCP segments without superuser privileges, so you can't just bring your own TCP congestion control algorithm in the userspace, unless you also wrap your custom TCP segments in UDP.
- bayindirh 1y ago
  Why do the hard work if the same thing can be done by the kernel, or even by the card itself? Drown other applications for your own benefit?
  - lxgr 1y ago
    > Why do the hard work if the same thing can be done by the kernel, or even by the card itself? How would you swap out the TCP congestion control algorithm in an OS, or even hardware, you don't control? > Drown other applications for your own benefit? Fairness equivalent to classic TCP is a design goal of practically all alternative algorithms, so I'm not sure what you're implying. It's entirely possible to improve responsivity without compromising on fairness, as e.g. BBR has shown.
    - neilalexander 1y ago
      > How would you swap out the TCP congestion control algorithm in an OS, or even hardware, you don't control? On the contrary, introducing their own novel/mysterious/poorly implemented congestion control algorithms is not a thing I want userspace applications doing.
      - lxgr 1y ago
        Fortunately you don't get any say in what my userspace applications do on my own hardware. And if you worry about hostile applications on your own hardware, the OS is an excellent point to limit what they can do – including overwhelming your network interface.
        neilalexander 1y ago
        > the OS is an excellent point to limit what they can do – including overwhelming your network interface One might even call this "congestion control"!
        lxgr 1y ago
        No, congestion control in the network sense is about congestion at choke points in packet networks due to higher inflow than outflow rates. While you're still on your own host, your OS has complete visibility into which applications are currently intending to send data and can schedule them much more directly.
  - jeffbee 1y ago
    "Drown other applications" is unfortunately exactly what happens when you let the Linux kernel run your TCP stack. Profile your application and you may discover that your CPUs are being spent running the protocol stack on behalf of other applications.
    - lxgr 1y ago
      What do you mean by "other applications"?
      - jeffbee 1y ago
        I mean when your application sends on a socket, the kernel may also send and receive traffic for another task while it's in the syscall, just for funsies, and this is true even if your applications are containerized and you believe their CPU cores are dedicated to the container.
        lxgr 1y ago
        Ah, but that's an OS implementation problem, not one with TCP or QUIC, no?
        jeffbee 1y ago
        Sure, but the ready appeal of QUIC is that it is in user space by nature, while Linux ties TCP to the kernel. You either need special privileges to run user space TCP on Linux, or you need a different operating system kernel altogether.

lxgr 1y ago

> QUIC’s design intentionally separates the wire protocol from the congestion control algorithm Is that not the case for TCP as well? Most congestion control algorithms just assign new meanings to existing wire-level flags (e.g. duplicate ACKs), or even only change sender-side behavior. > QUIC gives* control back to application developers to tailor congestion control to their use case* That's what it actually does: It moves the congestion control implementation from the OS to user space. In that sense, it's the same tradeoff as containers vs. regular binaries linking to shared libraries: Great if your applications are updated more often than your OS; not so great if it's the other way around.

bayindirh 1y ago
> QUIC gives control back to application developers to tailor congestion control to their use case If I understood modern application development correctly, this interprets as "The developers will import another library which they don't understand and will wreak havoc on other applications' data streams by only optimizing stuff for themselves". Again, if I remember correctly, an OS is "the layer which manages the sharing of limited resources among many processes which requests/needs it", and the OS can do system-wide, per socket congestion control without any effort because of the vantage point it has over networking layer. Assuming that every application will do congestion control correctly while not choking everyone else even unintentionally with user space's limited visibility is absurd at worst, and wishful thinking at best. The whole ordeal is direct violation with application separation coming with protected mode.
- lxgr 1y ago
  > Assuming that every application will do congestion control correctly while not choking everyone else even unintentionally with user space's limited visibility is absurd at worst, and wishful thinking at best. Why? All practically used variants of TCP achieve fairness at the bottleneck without any central arbiter or explicit view of the upstream congestion situation. Besides, UDP has never had congestion control and has been around for decades. > The whole ordeal is direct violation with application separation coming with protected mode. [...] The whole ordeal is direct violation with application separation coming with protected mode. Are you concerned about fairness between applications on the same OS or fairness at the bottleneck, usually upstream from both application and OS/host? In the former case, the OS always gets the last word, whether you're using TCP or your own homebrew congestion control via UDP, and you can make sure that each application gets the exact same fair egress rate if required. In the latter case, nothing prevents anyone from patching their kernel and running "unfair TCP" today.

IgorPartola 1y ago

The obnoxious thing is that overly aggressive firewalls have killed any IP protocols that are not TCP or UDP. Even ICMP is often blocked or partially blocked. In the mean time we could have had nice things: https://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr... SCTP would be a fantastic protocol for HTTP/HTTPS. Pipelining, multi homing, multi streaming, oh my.

Hydraulix989 1y ago
Not sure I follow. QUIC is just UDP with an extra header in the opaque payload. To a firewall, it looks just like UDP.
- jabiko 1y ago
  That's the point he is making. QUIC has to be based on UDP because the networking stack is ossified enough to not allow the addition of any new Layer 4 protocol. It's not a huge drawback though.
  - Hydraulix989 1y ago
    It's an eight byte penalty for things needed for QUIC anyways.
- p_l 1y ago
  It exists partially because of ossification of protocols that killed SCTP. As in, we wouldn't have to implement it on top of UDP
- sedawkgrep 1y ago
  I think they’re saying that due to how firewalls are deployed, everything end up either being built on tcp or udp, instead of using existing (or building new) layer four protocols more suited to solving the problem like sctp, et al. I’m not sure I agree though, because many firewalls already pass other protocols today, like GRE, IPSEC, etc.

jklowden 1y ago

Note well: the claims about TCP come with some evidence, in the form of a graph. The claims for QUIC do not. Many of the claims are dubious. TCP has "no notion of multiple steams"? What are two sockets, then? What is poll(2)? The onus is on QUIC to explain why it’s better for the application to multiplex the socket than for the kernel to multiplex the device. AFAICT that question is assumed away in a deluge of words. If the author thinks it’s the "end of TCP sockets", show us the research, the published papers and meticulous detail. Then tell me again why I should eschew the services of TCP and absorb its complexity into my application.

garganzol 1y ago
Manipulative tone of the article title says it all. "The end of".
- ashtakeaway 1y ago
  Really glad NextDNS blocked my click.
tuetuopay 1y ago
Even the TCP graph is dubious. Cubic being systematically above the link capacity makes me chuckle. Yes bufferbloat can have cubic "hug" a somewhat higher limit, but it still needs to start under the link capacity.
- lxgr 1y ago
  That's easily explained, in the parts of the x axis missing in the plot it actually goes negative and pays back the borrowed bytes.
cbluth 1y ago
The obvious comparisons are between udp and tcp, and then between quic and ssh, given the notion of "multiple streams"

superkuh 1y ago

QUIC would be the end of the free internet if it ever "took over" but luckily it won't. It's not built to do so, it's only built for corporate use cases. QUIC implementations do not allow for anyone to connect to anyone else. Instead, because it was built entirely with corporate for-profit uses cases in mind and open-washed through the IETF, the idea of a third party coporation having to authenticate the identity of all connections is baked in. And 99.999% of QUIC libs, and the way they're shipped in clients, cannot even connect to a server without a third party corp first saying they know the end point and allow it. Fine for corporate/profit use cases where security of the monetary transactions is all that matters. Very much less fine for human uses cases where it forces centralization and easy control by our rapidly enshittifying authoritarian governments. QUIC is the antithesis to the concept of the internet and it's robustness and routing around damage.

sleepydog 1y ago
I guess you are referring to the TLS requirement? I guess I could see how on a more restrictive platform like a phone you could conceivably be prevented from accepting alternate CAs or self signed certificates.
jauntywundrkind 1y ago
There's a fairly far a long draft for replacing webrtc's SCTP with QUIC for doing p2p work. It doesn't seem to have any of these challenges, seems to be perfectly viable there for connecting peers. https://github.com/w3c/p2p-webtransport https://github.com/w3c/p2p-webtransport Alas alas, basically stalled out, afaik no implementation. I wish Microsoft (the spec author) or someone would pick this back up.
- lxgr 1y ago
  WebRTC wraps SCTP in DTLS, so the "great challenge of encryption" has never been a problem there. It just uses self-signed certificates, which is maybe conceptually slightly clunky compared to "pure" anonymous TOFU, but allows reusing existing stacks.
lxgr 1y ago

KaiserPro 1y ago

One thing to note is that using HTTP2.0 for anything other than "this is not how to design high throughput protocols" is unfair. At the time HTTP2.0's multiplexing was known to be bad for anything other than perfect, low latency networks. I hope this was because people had faith in better connectivity, rather than ignorance of how mobile and non-lan traffic worked. You should probably at least try QUIC now, but you can get past HOL blocking by having multiple TCP streams. Its super cheap, cheaper than QUIC.

lxgr 1y ago
> you can get past HOL blocking by having multiple TCP streams. Its super cheap, cheaper than QUIC And also super inefficient, since it duplicates the TLS handshake across streams and uses more resources in the OS and middleboxes (like them or hate them, they're a thing that might throttle you if you go too crazy with connection-level parallelism). That's on top of very poor fairness at bottlenecks (which is per TCP stream unless there's separate traffic policing).
- KaiserPro 1y ago
  > since it duplicates the TLS handshake across streams Depends on your usecase of course. if its something thats only around for a minute, then yeah, but over an hour, makes no difference at all. QUIC eats more CPU per MB of data than plain TCP/TLS. but that might be solved by more modern offload/kernel optimisations
tuetuopay 1y ago
Even on perfect networks HoL blocking is an issue. If the receiving end of a set of stream blocks on one stream, it ends up blocking the whole connection. One stream stops due to backpressure, all streams stop.
- KaiserPro 1y ago
  Oh yeah, but thats why HTTP2.0 was flawed from the start.

exabrial 1y ago

The most obnoxious thing about QUIC is I don't need encryption all of the time, actually the majority of the time. Useless overhead.

lxgr 1y ago
Even if you have nothing to hide and don't care about accidental or intentional data modification, the benefit of largely cutting out "clever" middleboxes alone is almost always worth it.

Ingon 1y ago

I used QUIC extensively to implement https://github.com/connet-dev/connet https://github.com/connet-dev/connet and while I'm super happy with how it turned out, I think QUIC currently suffers from some immaturity - most implementations are still ongoing/not production ready (for example in java) and in many cases it is only viewed as a way to power on HTTP/3, instead of being self-standing protocol/API that ppl can use (for example, trying to use quic in android). In any case, I'm optimistic that QUIC has a bright future. I don't expect it to replace TCP, but give us another tool we can use when it is called for.

mannyv 1y ago

Leaving congestion control to apps means that congestion control at a system level will be impossible. Is that really a QUIC thing?

yencabulator 1y ago

The article starts with > Why QUIC’s user-space transport lets us ‘kill’ the old app-level event loop But then doesn't seem to mention that topic ever again. I don't see how QUIC changes that much.