7 ms·

In the early days of QUIC, many people pointed out that the UDP stack has had far far less optimization put into it than the TCP stack. Sure enough, some of the

In the early days of QUIC, many people pointed out that the UDP stack has had far far less optimization put into it than the TCP stack. Sure enough, some of the issues identified here arise because the UDP stack isn't doing things that it could do but that nobody has been motivated to make it do, such as UDP generic receive offload. Papers like this are very likely to lead to optimizations both obvious and subtle.

nextaccountic 2y ago
Do you mean that under the same workload, TCP will perform better?
Animats 2y ago
What is UDP offload going to do? UDP barely does anything but queue and copy. Linux scheduling from packet-received to thread has control is not real-time, and if the CPUs are busy, may be rather slow. That's probably part of the bottleneck. The embarrassing thing is that QUIC, even in Google's own benchmarks, only improved performance by about 10%. The added complexity probably isn't worth the trouble. However, it gave Google control of more of the stack, which may have been the real motivation.
- Vecr 2y ago
  I think one of the original drivers was the ability to quickly tweak parameters, after Linux rejected what I think was userspace adjustment of window sizing to be more aggressive than the default. The Linux maintainers didn't want to be responsible for congestion collapse, but UDP lets you spray packets from userspace, so Google went with that.
- JoshTriplett 2y ago
  Among other things, GRO (receive offloading) means you can get more data off of the network card in fewer operations. Linux has receive packet steering, which can help with getting packets from the network card to the right CPU and the right userspace thread without moving from one CPU's cache to another.
  - suprjami 2y ago
    RPS is just software RSS. You mean Receive Flow Steering, and RFS can only control RPS, so to do it in hardware you actually mean Accelerated RFS (which requires a pretty fancy NIC these days). Even ignoring the hardware requirement, unfortunately it's not that simple. I find results vary wildly whether you should put process and softirq on the same CPU core (sharing L1 and L2) or just on the same CPU socket (sharing L3 but don't constantly blow out L1/L2). Eric Dumazet said years ago at a Netdev.conf that L1 cache sizes have really not kept up with reality. That matches my experience. QUIC doing so much in userspace adds another class of application which has a so-far uncommon design pattern. I don't think it's possible to say whether any QUIC application benefits from RFS or not.
- infogulch 2y ago
  In my head the main benefit of QUIC was always multipath, aka the ability to switch interfaces on demand without losing the connection. There's MPTCP but who knows how viable it is.
  - rocqua 2y ago
    Mptcp sees use in the Telco space, so they probably know.
  - modeless 2y ago
    Is that actually implemented and working in practice? My connection still hangs whenever my wifi goes out of range...
  - Sesse__ 2y ago
    Apple's Siri is using MPTCP, so it is presumably viable.
    - jshier 2y ago
      It requires explicit backend support, and Apple supports it for many of their services, but I've never seen another public API that does. Anyone have any examples?
      - mh- 2y ago
        Last I looked into this (many years), ELB/GLBs didn't support it on AWS/GCP respectively. That prevented us from further considering implementing it at the time (mobile app -> AWS-hosted EC2 instances behind an ELB). Not sure if that's changed, but at the time it wasn't worth having to consider rolling our own LBs. To answer your original question, no, I haven't (knowingly) seen it on any public APIs.
  - suprjami 2y ago
    I always thought the main benefit of QUIC was to encrypt the important part of the transport header, so endpoints control their own destiny, not some middle device. If I had a dollar for every firewall vendor who thought dropping TCP retransmissions or TCP Reset was a good idea...
- amluto 2y ago
  Last I looked (several months ago), Linux's UDP stack did not seemed well tuned in its memory management accounting. For background, the mental model of what receiving network data looks like in userspace is almost completely backwards compared to how general-purpose kernel network receive actually works. User code thinks it allocates a buffer (per-socket or perhaps a fancier io_uring scheme), then receives packets into that buffer, then processes them. The kernel is the other way around. The kernel allocates buffers and feeds pointers to those buffers to the NIC. The NIC receives packets and DMAs them into the buffers, then tells the kernel. But the NIC and the kernel have absolutely no concept of which socket those buffers belong to until after they are DMAed into the buffers. So the kernel cannot possibly map received packets to the actual recipient's memory. So instead, after identifying who owns a received packet, the kernel retroactively charges the recipient for the memory. This happens on a per-packet basis, it involves per-socket and cgroup accounting, and there is no support for having a socket "pre-allocate" this memory in advance of receiving a packet. So the accounting is gnarly, involves atomic operations, and seems quite unlikely to win any speed awards. On a very cursory inspection, the TCP code seemed better tuned, and it possibly also won by generally handling more bytes per operation. Keep in mind that the kernel can't copy data to application memory synchronously -- the application memory might be paged out when a packet shows up. So instead the whole charging dance above happens immediately when a packet is received, and the data is copied later on. For quite a long time, I've thought it would be nifty if there was a NIC that kept received data in its own RAM and then allowed it to be efficiently DMAed to application memory when the application was ready for it. In essence, a lot of the accounting and memory management logic could move out of the kernel into the NIC. I'm not aware of anyone doing this.
  - fragmede 2y ago
    RDMA is common for high performance applications but it doesn't work over the Internet.
    - Danieru 2y ago
      It's a good thing the NIC is connected over pcie then.
    - 2y ago
- apitman 2y ago
  Ditching head of line blocking is potentially a big win, but I really wish it wouldn't have come with so much complexity.
- raggi 2y ago
  UDP offload gets you implicitly today: - 64 packets per syscall, which is enough data to amortize the syscall overhead - a single packet is not. - UDP offload optionally lets you defer checksum computation, often offloading it to hardware. - UDP offload lets you skip/reuse route lookups for subsequent packets in a bundle. What UDP offload is no good for though, is large scale servers - the current APIs only work when the incoming packet chains neatly organize into batches per peer socket. If you have many thousands of active sockets you’ll stop having full bundles and the overhead starts sneaking back in. As I said in another thread, we really need a replacement for the BSD APIs here, they just don’t scale for modern hardware constraints and software needs - much too expensive per packet.
- 10000truths 2y ago
  Bulk throughout isn't on par with TLS mainly because NICs with dedicated hardware for QUIC offload aren't commercially available (yet). Latency is undoubtedly better - the 1-RTT QUIC handshake substantially reduces time-to-first-byte compared to TLS.
- majke 2y ago
  > What is UDP offload going to do? Handling ACK packets in kernelspace would be one thing - helping for example RTT estimation. With userspace stack ACK's are handled in application and are subject to scheduler, suffering a lot on a loaded system.
  - morning-coffee 2y ago
    There are no ACKs inherent in the UDP protocol, so "UDP offload" is not where the savings are. There are ACKs in the QUIC protocol and they are carried by UDP datagrams which need to make their way up to user land to be processed, and this is the crux of the issue. What is needed is for QUIC offload to be invented/supported by HW so that most of the high-frequency/tiny-packet processing happens there, just as it does today for TCP offload. TCP large-send and large-receive offload is what is responsible for all the CPU savings as the application deals in 64KB or larger send/receives and the segmentation and receive coalescing all happen in hardware before an interrupt is even generated to involve the kernel, let alone userland.
RachelF 2y ago
Also bear in mind that many of today's network cards have processors in them that handle much of the TCP/IP overhead.
- kccqzy 2y ago
  That's mostly still for the data center. Which end-user network cards that I can buy can do TCP offloading?
  - phil21 2y ago
    Unless I’m missing something here, pretty much any Intel nic released in the past decade should support tcp offload. I imagine the same is true for Broadcom and other vendors as well, but I don’t have something handy to check.
  - JoshTriplett 2y ago
    Some wifi cards offload a surprising amount in order to do wake-on-wireless, but that's not for performance.
  - throw0101c 2y ago
    > Which end-user network cards that I can buy can do TCP offloading? Intel's I210 controllers support offloading: > Other performance-enhancing features include IPv4 and IPv6 checksum offload, TCP/UDP checksum offload, extended Tx descriptors for more offload capabilities, up to 256 KB TCP segmentation (TSO v2), header splitting, 40 KB packet buffer size, and 9.5 KB Jumbo Frame support. * https://cdrdv2-public.intel.com/327935/327935-Intel%20Ethernet%20Conroller-I210-product-brief.pdf https://cdrdv2-public.intel.com/327935/327935-Intel%20Ethern... And cost US$ 22: * https://www.amazon.com/dp/B0728289M7/ https://www.amazon.com/dp/B0728289M7/
  - vel0city 2y ago
    Practically every on-board network adapter I've had for over a decade has had TCP offload support. Even the network adapter on my cheap $300 Walmart laptop has hardware TCP offload support.
  - 2y ago
skywhopper 2y ago
The whole reason QUIC even exists in user space is because its developers were trying to hack a quick speed-up to HTTP rather than actually do the work to improve the underlying networking fundamentals. In this case the practicalities seem to have caught them out. If you want to build a better TCP, do it. But hacking one in on top of UDP was a cheat that didn’t pay off. Well, assuming performance was even the actual goal.
- osmarks 2y ago
  They couldn't have built it on anything but UDP because the world is now filled with poorly designed firewall/NAT middleboxes which will not route things other than TCP, UDP and optimistically ICMP.
- kbolino 2y ago
  It already exists, it's called SCTP. It doesn't work over the Internet because there's too much crufty hardware in the middle that will drop it instead of routing it. Also, Microsoft refused to implement it in Windows and also banned raw sockets so it's impossible to get support for it on that platform without custom drivers that practically nobody will install. I don't know how familiar the developers of QUIC were with SCTP in particular but they were definitely aware of the problems that prevented a better TCP from existing. The only practical solution is to build something on top of UDP, but if even that option proves unworkable, then the only other possibility left is to fragment the Internet.
  - suprjami 2y ago
    I like (some aspects of) SCTP too but it's not a solution to this problem. If you've followed Dave Taht's bufferbloat stuff, the reason he lost faith in TCP is because middle devices have access to the TCP header and can interfere with it. If SCTP got popular, then middle devices would ruin SCTP in the same way. QUIC is the bufferbloat preferred solution because the header is encrypted. It's not possible for a middle device to interfere with QUIC. Endpoints, and only endpoints, control their own traffic.
- adgjlsfhk1 2y ago
morning-coffee 2y ago
The UDP optimizations are already there and have been pretty much wrung out. https://www.fastly.com/blog/measuring-quic-vs-tcp-computational-efficiency/ https://www.fastly.com/blog/measuring-quic-vs-tcp-computatio... has good details and was done almost five years ago. The solution isn't in more UDP offload optimizations as there aren't any semantics in UDP that are expensive other than the quantity and frequency of datagrams to be processed in the context of the QUIC protocol that uses UDP as a transport. QUIC's state machine needs to see every UDP datagram carrying QUIC protocol messages in order to move forward. Just like was done for TCP offload more than twenty years ago, portions of QUIC state need to move and be maintained in hardware to prevent the host from having to see so many high-frequency tiny state updates messages.
suprjami 2y ago
Your first point is correct - papers ideally lead to innovation and tangible software improvements. I think a kernel implementation of QUIC is the next logical step. A context switch to decrypt a packet header and send control traffic is just dumb. That's the kernel's job. Userspace network stacks have never been a good idea. QUIC is no different. (edit: Xin Long already has started a kernel implementation, see elsewhere on this page)

nextaccountic 2y ago

Do you mean that under the same workload, TCP will perform better?

Animats 2y ago

What is UDP offload going to do? UDP barely does anything but queue and copy. Linux scheduling from packet-received to thread has control is not real-time, and if the CPUs are busy, may be rather slow. That's probably part of the bottleneck. The embarrassing thing is that QUIC, even in Google's own benchmarks, only improved performance by about 10%. The added complexity probably isn't worth the trouble. However, it gave Google control of more of the stack, which may have been the real motivation.

Vecr 2y ago
I think one of the original drivers was the ability to quickly tweak parameters, after Linux rejected what I think was userspace adjustment of window sizing to be more aggressive than the default. The Linux maintainers didn't want to be responsible for congestion collapse, but UDP lets you spray packets from userspace, so Google went with that.
JoshTriplett 2y ago
Among other things, GRO (receive offloading) means you can get more data off of the network card in fewer operations. Linux has receive packet steering, which can help with getting packets from the network card to the right CPU and the right userspace thread without moving from one CPU's cache to another.
- suprjami 2y ago
  RPS is just software RSS. You mean Receive Flow Steering, and RFS can only control RPS, so to do it in hardware you actually mean Accelerated RFS (which requires a pretty fancy NIC these days). Even ignoring the hardware requirement, unfortunately it's not that simple. I find results vary wildly whether you should put process and softirq on the same CPU core (sharing L1 and L2) or just on the same CPU socket (sharing L3 but don't constantly blow out L1/L2). Eric Dumazet said years ago at a Netdev.conf that L1 cache sizes have really not kept up with reality. That matches my experience. QUIC doing so much in userspace adds another class of application which has a so-far uncommon design pattern. I don't think it's possible to say whether any QUIC application benefits from RFS or not.
infogulch 2y ago
In my head the main benefit of QUIC was always multipath, aka the ability to switch interfaces on demand without losing the connection. There's MPTCP but who knows how viable it is.
- rocqua 2y ago
  Mptcp sees use in the Telco space, so they probably know.
- modeless 2y ago
  Is that actually implemented and working in practice? My connection still hangs whenever my wifi goes out of range...
- Sesse__ 2y ago
  Apple's Siri is using MPTCP, so it is presumably viable.
  - jshier 2y ago
    It requires explicit backend support, and Apple supports it for many of their services, but I've never seen another public API that does. Anyone have any examples?
    - mh- 2y ago
      Last I looked into this (many years), ELB/GLBs didn't support it on AWS/GCP respectively. That prevented us from further considering implementing it at the time (mobile app -> AWS-hosted EC2 instances behind an ELB). Not sure if that's changed, but at the time it wasn't worth having to consider rolling our own LBs. To answer your original question, no, I haven't (knowingly) seen it on any public APIs.
- suprjami 2y ago
  I always thought the main benefit of QUIC was to encrypt the important part of the transport header, so endpoints control their own destiny, not some middle device. If I had a dollar for every firewall vendor who thought dropping TCP retransmissions or TCP Reset was a good idea...
amluto 2y ago
Last I looked (several months ago), Linux's UDP stack did not seemed well tuned in its memory management accounting. For background, the mental model of what receiving network data looks like in userspace is almost completely backwards compared to how general-purpose kernel network receive actually works. User code thinks it allocates a buffer (per-socket or perhaps a fancier io_uring scheme), then receives packets into that buffer, then processes them. The kernel is the other way around. The kernel allocates buffers and feeds pointers to those buffers to the NIC. The NIC receives packets and DMAs them into the buffers, then tells the kernel. But the NIC and the kernel have absolutely no concept of which socket those buffers belong to until after they are DMAed into the buffers. So the kernel cannot possibly map received packets to the actual recipient's memory. So instead, after identifying who owns a received packet, the kernel retroactively charges the recipient for the memory. This happens on a per-packet basis, it involves per-socket and cgroup accounting, and there is no support for having a socket "pre-allocate" this memory in advance of receiving a packet. So the accounting is gnarly, involves atomic operations, and seems quite unlikely to win any speed awards. On a very cursory inspection, the TCP code seemed better tuned, and it possibly also won by generally handling more bytes per operation. Keep in mind that the kernel can't copy data to application memory synchronously -- the application memory might be paged out when a packet shows up. So instead the whole charging dance above happens immediately when a packet is received, and the data is copied later on. For quite a long time, I've thought it would be nifty if there was a NIC that kept received data in its own RAM and then allowed it to be efficiently DMAed to application memory when the application was ready for it. In essence, a lot of the accounting and memory management logic could move out of the kernel into the NIC. I'm not aware of anyone doing this.
- fragmede 2y ago
  RDMA is common for high performance applications but it doesn't work over the Internet.
  - Danieru 2y ago
    It's a good thing the NIC is connected over pcie then.
  - 2y ago
apitman 2y ago
Ditching head of line blocking is potentially a big win, but I really wish it wouldn't have come with so much complexity.
raggi 2y ago
UDP offload gets you implicitly today: - 64 packets per syscall, which is enough data to amortize the syscall overhead - a single packet is not. - UDP offload optionally lets you defer checksum computation, often offloading it to hardware. - UDP offload lets you skip/reuse route lookups for subsequent packets in a bundle. What UDP offload is no good for though, is large scale servers - the current APIs only work when the incoming packet chains neatly organize into batches per peer socket. If you have many thousands of active sockets you’ll stop having full bundles and the overhead starts sneaking back in. As I said in another thread, we really need a replacement for the BSD APIs here, they just don’t scale for modern hardware constraints and software needs - much too expensive per packet.
10000truths 2y ago
Bulk throughout isn't on par with TLS mainly because NICs with dedicated hardware for QUIC offload aren't commercially available (yet). Latency is undoubtedly better - the 1-RTT QUIC handshake substantially reduces time-to-first-byte compared to TLS.
majke 2y ago
> What is UDP offload going to do? Handling ACK packets in kernelspace would be one thing - helping for example RTT estimation. With userspace stack ACK's are handled in application and are subject to scheduler, suffering a lot on a loaded system.
- morning-coffee 2y ago
  There are no ACKs inherent in the UDP protocol, so "UDP offload" is not where the savings are. There are ACKs in the QUIC protocol and they are carried by UDP datagrams which need to make their way up to user land to be processed, and this is the crux of the issue. What is needed is for QUIC offload to be invented/supported by HW so that most of the high-frequency/tiny-packet processing happens there, just as it does today for TCP offload. TCP large-send and large-receive offload is what is responsible for all the CPU savings as the application deals in 64KB or larger send/receives and the segmentation and receive coalescing all happen in hardware before an interrupt is even generated to involve the kernel, let alone userland.

RachelF 2y ago

Also bear in mind that many of today's network cards have processors in them that handle much of the TCP/IP overhead.

kccqzy 2y ago
That's mostly still for the data center. Which end-user network cards that I can buy can do TCP offloading?
- phil21 2y ago
  Unless I’m missing something here, pretty much any Intel nic released in the past decade should support tcp offload. I imagine the same is true for Broadcom and other vendors as well, but I don’t have something handy to check.
- JoshTriplett 2y ago
  Some wifi cards offload a surprising amount in order to do wake-on-wireless, but that's not for performance.
- throw0101c 2y ago
  > Which end-user network cards that I can buy can do TCP offloading? Intel's I210 controllers support offloading: > Other performance-enhancing features include IPv4 and IPv6 checksum offload, TCP/UDP checksum offload, extended Tx descriptors for more offload capabilities, up to 256 KB TCP segmentation (TSO v2), header splitting, 40 KB packet buffer size, and 9.5 KB Jumbo Frame support. * https://cdrdv2-public.intel.com/327935/327935-Intel%20Ethernet%20Conroller-I210-product-brief.pdf https://cdrdv2-public.intel.com/327935/327935-Intel%20Ethern... And cost US$ 22: * https://www.amazon.com/dp/B0728289M7/ https://www.amazon.com/dp/B0728289M7/
- vel0city 2y ago
  Practically every on-board network adapter I've had for over a decade has had TCP offload support. Even the network adapter on my cheap $300 Walmart laptop has hardware TCP offload support.
- 2y ago

skywhopper 2y ago

The whole reason QUIC even exists in user space is because its developers were trying to hack a quick speed-up to HTTP rather than actually do the work to improve the underlying networking fundamentals. In this case the practicalities seem to have caught them out. If you want to build a better TCP, do it. But hacking one in on top of UDP was a cheat that didn’t pay off. Well, assuming performance was even the actual goal.

osmarks 2y ago
They couldn't have built it on anything but UDP because the world is now filled with poorly designed firewall/NAT middleboxes which will not route things other than TCP, UDP and optimistically ICMP.
kbolino 2y ago
It already exists, it's called SCTP. It doesn't work over the Internet because there's too much crufty hardware in the middle that will drop it instead of routing it. Also, Microsoft refused to implement it in Windows and also banned raw sockets so it's impossible to get support for it on that platform without custom drivers that practically nobody will install. I don't know how familiar the developers of QUIC were with SCTP in particular but they were definitely aware of the problems that prevented a better TCP from existing. The only practical solution is to build something on top of UDP, but if even that option proves unworkable, then the only other possibility left is to fragment the Internet.
- suprjami 2y ago
  I like (some aspects of) SCTP too but it's not a solution to this problem. If you've followed Dave Taht's bufferbloat stuff, the reason he lost faith in TCP is because middle devices have access to the TCP header and can interfere with it. If SCTP got popular, then middle devices would ruin SCTP in the same way. QUIC is the bufferbloat preferred solution because the header is encrypted. It's not possible for a middle device to interfere with QUIC. Endpoints, and only endpoints, control their own traffic.
adgjlsfhk1 2y ago

morning-coffee 2y ago

The UDP optimizations are already there and have been pretty much wrung out. https://www.fastly.com/blog/measuring-quic-vs-tcp-computational-efficiency/ https://www.fastly.com/blog/measuring-quic-vs-tcp-computatio... has good details and was done almost five years ago. The solution isn't in more UDP offload optimizations as there aren't any semantics in UDP that are expensive other than the quantity and frequency of datagrams to be processed in the context of the QUIC protocol that uses UDP as a transport. QUIC's state machine needs to see every UDP datagram carrying QUIC protocol messages in order to move forward. Just like was done for TCP offload more than twenty years ago, portions of QUIC state need to move and be maintained in hardware to prevent the host from having to see so many high-frequency tiny state updates messages.

suprjami 2y ago

Your first point is correct - papers ideally lead to innovation and tangible software improvements. I think a kernel implementation of QUIC is the next logical step. A context switch to decrypt a packet header and send control traffic is just dumb. That's the kernel's job. Userspace network stacks have never been a good idea. QUIC is no different. (edit: Xin Long already has started a kernel implementation, see elsewhere on this page)