6 ms·

Disabling TCP offload to hardware makes this comparison much less useful. TCP has hardware offload right now, and QUIC will take years to get at that level.

drewg123 6y ago
Indeed. Given that the protocol itself is encrypted, doing things like TCP Large Send (TSO) in hardware gets much trickier.
- emmericp 6y ago
  Isn't a large part of the performance gain from TSO offloading on transmit due to avoiding working with way too many skb's in the kernel? Shouldn't GSO for UDP with proper NIC support give you most of the performance gain? I'm not sure about the implementation details of GSO and TSO in Linux. But I know a thing or two about the hardware: the Intel Niantic/82599 NICs support UDP transmit segmentation offloading, I've checked the datasheet real quick and it says "Note that current UDP segmentation offload is not supported by any standard OS." Not sure if this is just an outdated comment in a 10 year old datasheet or if there's some performance to be gained by using this UDP segmentation offload for quick. Edit: Should have clicked on the link in the article [1] before posting here, UDP segementation offloading is coming to lots of NIC drivers in 5.5. Always interesting how the hardware often supports way more features than the drivers expose and you suddenly get new hardware features on a 10+ year old NIC model :) (Similar story with IPsec offloading on the Niantic/82599) [1] https://kernelnewbies.org/Linux_5.5#Networking-1 https://kernelnewbies.org/Linux_5.5#Networking-1
  - richardwhiuk 6y ago
    TSO and GSO improve network bandwidth, mainly.
telotortium 6y ago
How much does TCP offload help in practice, in particular for TLS connections? Is it a 20% improvement, a 2X improvement, etc.? That would be good to know to evaluate the performance of QUIC in the article.
- unethical_ban 6y ago
  I don't understand how TLS makes a difference. They are different layers of the stack.
  - gok 6y ago
    I think the question is: given that you're going to be performing pretty heavy symmetric encryption on every byte anyway, how much can TCP offload really help when all it's handling is the relatively trivial checksum and sliding window computation?
  - geocar 6y ago
    Having 40Gb/sec TCP is great but if you can't come up with 40Gb/sec to put into it, "what's the point?" Of course, you could just buy a TLS accelerator if you wanted one.
    - ComputerGuru 6y ago
      CPU utilization is a zero-sum game. If you need to use the CPU to calculate checksums or do packet segmentation then it’s that much less CPU available for TLS.
      - alfalfasprout 6y ago
        So not quite... a lot of modern CPUs have crypto primitives implemented in hardware that allow other instructions to run simultaneously on the same core (taking advantage of hyperthreading and out of order execution).
      - geocar 6y ago
        That's one way to look at it. Another way is if it takes a couple CPUs to run my application, a CPU or two to drive the storage, and a CPU to do TLS, why would I also spend a CPU on TCP if my network card already has one that does that? That's a little bit like what programmers are talking about when they are talking about "multi-core programming", except those are general-purpose "cores", and (presumably) these "cores" can only handle doing TCP.
PudgePacket 6y ago
If I understand correctly the whole point of QUIC is to not be in hardware level. That's why QUIC exists, because TCP can't be changed because it is baked in at such low, fundamental levels.
- oneplane 6y ago
  No, the point of QUIC is to shave a few % of traffic off of the loads the likes of FAANG have, saving millions, while having a marginal benefit to pretty much everyone else. QUIC can be 'a bit better' in 'some scenarios', but most of the time it is totally irrelevant, except for the fact that it requires TLS 1.3 which is a good requirement overall.
  - jeffbee 6y ago
    QUIC in the datacenter is hugely superior to Linux kernel TCP. Kernel TCP has been inappropriate for datacenter networking for decades and this has been a known issue mentioned many times in the literature (start with [1] if this is new to you). Datacenter operators have been forced to either patch the kernel or bypass it to get decent TCP performance. QUIC is a rational response to this situation. Instead of waiting a quarter century or more for Linux to get its TCP stack in order, switch to UDP and innovate in user space. 1: https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf
    - derefr 6y ago
      Are you talking about TCP over datacenter hardware Ethernet, or TCP over a software-defined network (potentially terminated on the same host)?
      - jeffbee 6y ago
        Not sure these are opposing topics. My understanding of SDN is simply that a program may decide to drop certain frames, and this can run on top of ordinary Ethernet.
toast0 6y ago
I'm much less worried about tcp segmentation offload being disabled, than a single core, single threaded test. QUIC is going to hurt performance the most on multicore, multithreaded loads, where with TCP, you're getting the NIC to hash packets and deliver them to per-cpu rxqueues, and (hopefully) avoiding a lot of cross-cpu locking. That's going to be painful on QUIC unless/until you can get NICs to hash on the connection ids.
- jeffbee 6y ago
  Having a receive queue on each CPU is usually counterproductive in my experience ... one per longest-latency-cache domain is better in most cases. Anyway these days many NICs multiqueue resources have been outstripped by the number of cores in the box. We have NICs with 32 or 64 queues in boxes having 88 or 128 CPUs. RFS flow tables are installed by sendmsg and recvmsg, so I fail to see why RFS doesn't benefit UDP just as much as TCP. If you are using RFS, your QUIC packets should still arrive on the correct CPU.
  - derefr 6y ago
    > We have NICs with 32 or 64 queues in boxes having 88 or 128 CPUs. Doesn't that mean that you should have more NICs? From my recollection, the main thing that's made a computer a server or mainframe for the last 50 years has been that such systems have enough IO channels to saturate the CPU (i.e. they're intended to be CPU-bound, not IO-bound, under heavily-concurrent workloads.) If you take all the IO-offload cards out of a mainframe, it's just a regular computer.
    - xxpor 6y ago
      PCIe Gen 3 is becoming a real problem here. There's only so many lanes you get per socket, it's completely possible today to end up being IO bound just due to PCIe bandwidth, not network speeds or CPU processing power. For example, this Cascade Lake Xeon has 48 lanes of PCIe gen 3: https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=192458 https://ark.intel.com/content/www/us/en/ark/compare.html?pro... If you give yourself 32 of 48 lanes for networking, that's only 256 gbits of PCIe bandwidth. So each 16x slot is limited to a 100 gbit card. And that's a 22 core processor. With DPDK, coming up with a application that could process 200 gbit is almost trivial.
petethepig 6y ago
Is that common in consumer or even server hardware though?
- danudey 6y ago
  Yes, quite. The i219 chipset in my laptop "includes advanced interrupt handling features to reduce CPU overheard. Other performance-enhancing features include offloading TCP/UDP (for both IPv4 and IPv6) checksum calculations and performing TCP segmentation. Advanced features such as Jumbo Frame support for extra-large packets and Receive Side Scaling (RSS) are also supported." It's also supported on the old chipset from the motherboard I bought more than half a decade ago, and every chipset I've used in the server space in as long as I can remember.
alfalfasprout 6y ago
With Intel DPDK now in a fairly good state, it's actually fairly possible to get very attractive performance in userspace. You can leverage CPU instructions for crypto primitives when available to speed things up too. The amount of TCP hardware offload in linux right now is actually fairly limited. Kernel developers have largely rejected true in-hardware TCP stacks for a while now (although you can get this functionality with high end NICs and custom kernel builds). That said, since QUIC generally reduces the ack frequency it should be possible to match or exceed TCP with hardware offload. Especially when you look at total throughput with many streams open.

drewg123 6y ago

Indeed. Given that the protocol itself is encrypted, doing things like TCP Large Send (TSO) in hardware gets much trickier.

emmericp 6y ago
Isn't a large part of the performance gain from TSO offloading on transmit due to avoiding working with way too many skb's in the kernel? Shouldn't GSO for UDP with proper NIC support give you most of the performance gain? I'm not sure about the implementation details of GSO and TSO in Linux. But I know a thing or two about the hardware: the Intel Niantic/82599 NICs support UDP transmit segmentation offloading, I've checked the datasheet real quick and it says "Note that current UDP segmentation offload is not supported by any standard OS." Not sure if this is just an outdated comment in a 10 year old datasheet or if there's some performance to be gained by using this UDP segmentation offload for quick. Edit: Should have clicked on the link in the article [1] before posting here, UDP segementation offloading is coming to lots of NIC drivers in 5.5. Always interesting how the hardware often supports way more features than the drivers expose and you suddenly get new hardware features on a 10+ year old NIC model :) (Similar story with IPsec offloading on the Niantic/82599) [1] https://kernelnewbies.org/Linux_5.5#Networking-1 https://kernelnewbies.org/Linux_5.5#Networking-1
- richardwhiuk 6y ago
  TSO and GSO improve network bandwidth, mainly.

telotortium 6y ago

How much does TCP offload help in practice, in particular for TLS connections? Is it a 20% improvement, a 2X improvement, etc.? That would be good to know to evaluate the performance of QUIC in the article.

unethical_ban 6y ago
I don't understand how TLS makes a difference. They are different layers of the stack.
- gok 6y ago
  I think the question is: given that you're going to be performing pretty heavy symmetric encryption on every byte anyway, how much can TCP offload really help when all it's handling is the relatively trivial checksum and sliding window computation?
- geocar 6y ago
  Having 40Gb/sec TCP is great but if you can't come up with 40Gb/sec to put into it, "what's the point?" Of course, you could just buy a TLS accelerator if you wanted one.
  - ComputerGuru 6y ago
    CPU utilization is a zero-sum game. If you need to use the CPU to calculate checksums or do packet segmentation then it’s that much less CPU available for TLS.
    - alfalfasprout 6y ago
      So not quite... a lot of modern CPUs have crypto primitives implemented in hardware that allow other instructions to run simultaneously on the same core (taking advantage of hyperthreading and out of order execution).
    - geocar 6y ago
      That's one way to look at it. Another way is if it takes a couple CPUs to run my application, a CPU or two to drive the storage, and a CPU to do TLS, why would I also spend a CPU on TCP if my network card already has one that does that? That's a little bit like what programmers are talking about when they are talking about "multi-core programming", except those are general-purpose "cores", and (presumably) these "cores" can only handle doing TCP.

PudgePacket 6y ago

If I understand correctly the whole point of QUIC is to not be in hardware level. That's why QUIC exists, because TCP can't be changed because it is baked in at such low, fundamental levels.

oneplane 6y ago
No, the point of QUIC is to shave a few % of traffic off of the loads the likes of FAANG have, saving millions, while having a marginal benefit to pretty much everyone else. QUIC can be 'a bit better' in 'some scenarios', but most of the time it is totally irrelevant, except for the fact that it requires TLS 1.3 which is a good requirement overall.
- jeffbee 6y ago
  QUIC in the datacenter is hugely superior to Linux kernel TCP. Kernel TCP has been inappropriate for datacenter networking for decades and this has been a known issue mentioned many times in the literature (start with [1] if this is new to you). Datacenter operators have been forced to either patch the kernel or bypass it to get decent TCP performance. QUIC is a rational response to this situation. Instead of waiting a quarter century or more for Linux to get its TCP stack in order, switch to UDP and innovate in user space. 1: https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf
  - derefr 6y ago
    Are you talking about TCP over datacenter hardware Ethernet, or TCP over a software-defined network (potentially terminated on the same host)?
    - jeffbee 6y ago
      Not sure these are opposing topics. My understanding of SDN is simply that a program may decide to drop certain frames, and this can run on top of ordinary Ethernet.

toast0 6y ago

I'm much less worried about tcp segmentation offload being disabled, than a single core, single threaded test. QUIC is going to hurt performance the most on multicore, multithreaded loads, where with TCP, you're getting the NIC to hash packets and deliver them to per-cpu rxqueues, and (hopefully) avoiding a lot of cross-cpu locking. That's going to be painful on QUIC unless/until you can get NICs to hash on the connection ids.

jeffbee 6y ago
Having a receive queue on each CPU is usually counterproductive in my experience ... one per longest-latency-cache domain is better in most cases. Anyway these days many NICs multiqueue resources have been outstripped by the number of cores in the box. We have NICs with 32 or 64 queues in boxes having 88 or 128 CPUs. RFS flow tables are installed by sendmsg and recvmsg, so I fail to see why RFS doesn't benefit UDP just as much as TCP. If you are using RFS, your QUIC packets should still arrive on the correct CPU.
- derefr 6y ago
  > We have NICs with 32 or 64 queues in boxes having 88 or 128 CPUs. Doesn't that mean that you should have more NICs? From my recollection, the main thing that's made a computer a server or mainframe for the last 50 years has been that such systems have enough IO channels to saturate the CPU (i.e. they're intended to be CPU-bound, not IO-bound, under heavily-concurrent workloads.) If you take all the IO-offload cards out of a mainframe, it's just a regular computer.
  - xxpor 6y ago
    PCIe Gen 3 is becoming a real problem here. There's only so many lanes you get per socket, it's completely possible today to end up being IO bound just due to PCIe bandwidth, not network speeds or CPU processing power. For example, this Cascade Lake Xeon has 48 lanes of PCIe gen 3: https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=192458 https://ark.intel.com/content/www/us/en/ark/compare.html?pro... If you give yourself 32 of 48 lanes for networking, that's only 256 gbits of PCIe bandwidth. So each 16x slot is limited to a 100 gbit card. And that's a 22 core processor. With DPDK, coming up with a application that could process 200 gbit is almost trivial.

petethepig 6y ago

Is that common in consumer or even server hardware though?

danudey 6y ago
Yes, quite. The i219 chipset in my laptop "includes advanced interrupt handling features to reduce CPU overheard. Other performance-enhancing features include offloading TCP/UDP (for both IPv4 and IPv6) checksum calculations and performing TCP segmentation. Advanced features such as Jumbo Frame support for extra-large packets and Receive Side Scaling (RSS) are also supported." It's also supported on the old chipset from the motherboard I bought more than half a decade ago, and every chipset I've used in the server space in as long as I can remember.

alfalfasprout 6y ago

With Intel DPDK now in a fairly good state, it's actually fairly possible to get very attractive performance in userspace. You can leverage CPU instructions for crypto primitives when available to speed things up too. The amount of TCP hardware offload in linux right now is actually fairly limited. Kernel developers have largely rejected true in-hardware TCP stacks for a while now (although you can get this functionality with high end NICs and custom kernel builds). That said, since QUIC generally reduces the ack frequency it should be possible to match or exceed TCP with hardware offload. Especially when you look at total throughput with many streams open.