4 ms·

> Another problem is that it's CPU intensive. Are there 10GigE cards that do not do things like IP/TCP offloading at this point? Offloading dates back to (at

> Another problem is that it's CPU intensive. Are there 10GigE cards that do not do things like IP/TCP offloading at this point? Offloading dates back to (at least) 2005: * https://www.chelsio.com/independent-research-shows-10g-ethernet-with-tcp-offload-outperforms-infiniband-and-myrinet-1121720400/ https://www.chelsio.com/independent-research-shows-10g-ether... * https://www.networkworld.com/article/2312690/tcp-offload-lifts-10g-ethernet-over-other-technologies.html https://www.networkworld.com/article/2312690/tcp-offload-lif...

dale_glass 4y ago
You can go fast if you don't do anything fancy with the interface. If you say, want bridged networking for your VMs and add your 10G interface to virbr0, poof, a good chunk of your acceleration vanishes right there. Routing and firewalling also cost you a lot. There are ways to deal with this with eg, virtual functions, but the point is that even on modern hardware, 10G can be no longer a foolproof thing to have working at full capacity. You may need to actually do a fair amount of tweaking to have things perform well.
iptrans 4y ago
TCP/IP offload isn’t the issue. The core problem is that the Linux kernel uses interrupts for handling packets. This limits Linux networking performance in terms of packets per second. The limit is about a million packets per second per core. For reference 10GE is about 16 million packets per second at line rate using small packets. This is why you have to use kernel bypass software in user space to get linerate performance above 10G in Linux. Popular software for this use case utilize DPDK, XDP or VPP.
- toast0 4y ago
  You don't need an interrupt per packet, at least not with sensible NICs and OSes. Something like 10k interrupts per second is good enough, pick up a bunch of packets on each interrupt; you do lose out slightly on latency, but gain a lot of throughput. Look up 'interrupt moderation', it's not new, and most cards should support it. Professionlly, I ran dual xeon 2690v1 or v2 to 9Gbps for https download on FreeBSD; http hit 10G (only had one 10G to the internet on those machines), but crypto took too much CPU. Dual Xeon 2690v4 ran to 20Gbps, no problem (2x 14 core broadwell, much better AES acceleration, faster ram, more cores, etc, had dual 10G to the internet). Personally, I've just setup 10G between my two home servers, and can only manage about 5-8Gbps with iperf3, but that's with a pentium g2020 on one end (dual core Ivy Bridge, 10 years old at this point), and the network cards are configured for bridging, which means no tcp offloading. Edit: also, check out what Netflix has been doing with 800Gbps, although sendfile and TLS in the kernel cuts out a lot of userspace, kind of equal but opposite of cutting out kernelspace, http://nabstreamingsummit.com/wp-content/uploads/2022/05/2022-Streaming-Summit-Netflix.pdf http://nabstreamingsummit.com/wp-content/uploads/2022/05/202...
  - iptrans 4y ago
    Interrupt moderation only gives a modest improvement, as can be seen from the benchmarking done by Intel. Intel would also not have gone through the effort to develop DPDK if all you had to do to achieve linerate performance would be to enable interrupt moderation. Furthermore, quoting Gbps numbers is beside the point when the limiting factor is packets per second. It is trivial to improve Gbps numbers simply by using larger packets.
    - toast0 4y ago
      I'm quoting bulk transfer, with 1500 MTU. I could run jumbo packets for my internal network test and probably get better numbers, but jumbo packets are hard. When I was quoting https download on public internet, that pretty much means MTU 1500 as well, but was definitely the case. If you're sending smaller packets, sure, that's harder. I guess that's a big deal if you're a DNS server, or voip (audio only); but if you're doing any sort of bulk transfer, you're getting large enough packets. > Intel would also not have gone through the effort to develop DPDK if all you had to do to achieve linerate performance would be to enable interrupt moderation. DPDK has uses, sure. But you don't need it for 10G on decent hardware, which includes 7 year old server chips, if you're just doing bulk transfer.
      - iptrans 4y ago
        Bulk transfers aren’t that being interesting from a networking perspective. You gonna have a bad time if you optimize only for the best case scenario. Even using IMIX is a low bar. The proper way to do things is linerate using small packets.
- jabl 4y ago
  Most Linux network drives support NAPI since a couple of decades. No panacea of course, but still, far from having one interrupt per packet.
jandrese 4y ago
The other issue is that unless your computer is acting as a router or a bridge, you need to do something with that 10GB data stream. SSDs have only recently gotten fast enough to just barely support reading or writing that fast. But even if you do find one that supports writes that fast a 10GbeE card could fill an expensive 4TB drive in less than an hour. Good luck decoding JPEGs and blitting them out to a web browser window that fast.
- Dalewyn 4y ago
  >10GB data stream. SSDs have only recently gotten fast enough to just barely support reading or writing that fast. 10gbps (gigabits per second) is not 10GB/s (gigabytes per second). Specifically, 10gbps is approximately 1.25GB/s or 1250MB/s.
  - jandrese 4y ago
    Consumer SSDs used to max out at about 550MB/s, some still do. You need a larger and more modern drive to do 1.25GB/s sustained write. Even then buffering can get you.
    - Dalewyn 4y ago
      That's due to the communication protocol. 2.5 inch and M.2 SATA SSDs max out around 550MB/s due to the limits of SATA3 connections which cap out at 6gbps. M.2 NVME SSDs meanwhile communicate over PCIE, generally using four lanes, and the latest PCIE5 SSDs can do around 15GB/s if I recall. PCIE4 drives can get up to around 7GB/s, and PCIE3 drives up to around 3GB/s. Other potential bottlenecks can occur with the motherboard chipset, controller, and NAND flash, but details. TL;DR: Any NVME SSD can saturate a 10gbps ethernet connection.

dale_glass 4y ago

You can go fast if you don't do anything fancy with the interface. If you say, want bridged networking for your VMs and add your 10G interface to virbr0, poof, a good chunk of your acceleration vanishes right there. Routing and firewalling also cost you a lot. There are ways to deal with this with eg, virtual functions, but the point is that even on modern hardware, 10G can be no longer a foolproof thing to have working at full capacity. You may need to actually do a fair amount of tweaking to have things perform well.

iptrans 4y ago

TCP/IP offload isn’t the issue. The core problem is that the Linux kernel uses interrupts for handling packets. This limits Linux networking performance in terms of packets per second. The limit is about a million packets per second per core. For reference 10GE is about 16 million packets per second at line rate using small packets. This is why you have to use kernel bypass software in user space to get linerate performance above 10G in Linux. Popular software for this use case utilize DPDK, XDP or VPP.

toast0 4y ago
You don't need an interrupt per packet, at least not with sensible NICs and OSes. Something like 10k interrupts per second is good enough, pick up a bunch of packets on each interrupt; you do lose out slightly on latency, but gain a lot of throughput. Look up 'interrupt moderation', it's not new, and most cards should support it. Professionlly, I ran dual xeon 2690v1 or v2 to 9Gbps for https download on FreeBSD; http hit 10G (only had one 10G to the internet on those machines), but crypto took too much CPU. Dual Xeon 2690v4 ran to 20Gbps, no problem (2x 14 core broadwell, much better AES acceleration, faster ram, more cores, etc, had dual 10G to the internet). Personally, I've just setup 10G between my two home servers, and can only manage about 5-8Gbps with iperf3, but that's with a pentium g2020 on one end (dual core Ivy Bridge, 10 years old at this point), and the network cards are configured for bridging, which means no tcp offloading. Edit: also, check out what Netflix has been doing with 800Gbps, although sendfile and TLS in the kernel cuts out a lot of userspace, kind of equal but opposite of cutting out kernelspace, http://nabstreamingsummit.com/wp-content/uploads/2022/05/2022-Streaming-Summit-Netflix.pdf http://nabstreamingsummit.com/wp-content/uploads/2022/05/202...
- iptrans 4y ago
  Interrupt moderation only gives a modest improvement, as can be seen from the benchmarking done by Intel. Intel would also not have gone through the effort to develop DPDK if all you had to do to achieve linerate performance would be to enable interrupt moderation. Furthermore, quoting Gbps numbers is beside the point when the limiting factor is packets per second. It is trivial to improve Gbps numbers simply by using larger packets.
  - toast0 4y ago
    I'm quoting bulk transfer, with 1500 MTU. I could run jumbo packets for my internal network test and probably get better numbers, but jumbo packets are hard. When I was quoting https download on public internet, that pretty much means MTU 1500 as well, but was definitely the case. If you're sending smaller packets, sure, that's harder. I guess that's a big deal if you're a DNS server, or voip (audio only); but if you're doing any sort of bulk transfer, you're getting large enough packets. > Intel would also not have gone through the effort to develop DPDK if all you had to do to achieve linerate performance would be to enable interrupt moderation. DPDK has uses, sure. But you don't need it for 10G on decent hardware, which includes 7 year old server chips, if you're just doing bulk transfer.
    - iptrans 4y ago
      Bulk transfers aren’t that being interesting from a networking perspective. You gonna have a bad time if you optimize only for the best case scenario. Even using IMIX is a low bar. The proper way to do things is linerate using small packets.
jabl 4y ago
Most Linux network drives support NAPI since a couple of decades. No panacea of course, but still, far from having one interrupt per packet.

jandrese 4y ago

The other issue is that unless your computer is acting as a router or a bridge, you need to do something with that 10GB data stream. SSDs have only recently gotten fast enough to just barely support reading or writing that fast. But even if you do find one that supports writes that fast a 10GbeE card could fill an expensive 4TB drive in less than an hour. Good luck decoding JPEGs and blitting them out to a web browser window that fast.

Dalewyn 4y ago
>10GB data stream. SSDs have only recently gotten fast enough to just barely support reading or writing that fast. 10gbps (gigabits per second) is not 10GB/s (gigabytes per second). Specifically, 10gbps is approximately 1.25GB/s or 1250MB/s.
- jandrese 4y ago
  Consumer SSDs used to max out at about 550MB/s, some still do. You need a larger and more modern drive to do 1.25GB/s sustained write. Even then buffering can get you.
  - Dalewyn 4y ago
    That's due to the communication protocol. 2.5 inch and M.2 SATA SSDs max out around 550MB/s due to the limits of SATA3 connections which cap out at 6gbps. M.2 NVME SSDs meanwhile communicate over PCIE, generally using four lanes, and the latest PCIE5 SSDs can do around 15GB/s if I recall. PCIE4 drives can get up to around 7GB/s, and PCIE3 drives up to around 3GB/s. Other potential bottlenecks can occur with the motherboard chipset, controller, and NAND flash, but details. TL;DR: Any NVME SSD can saturate a 10gbps ethernet connection.