3 ms·

I mean, larger packets (and working path MTU detection) could be useful, but with large (1500) byte packets and reasonable hardware, I never had trouble pushing

by toast0 2y ago

I mean, larger packets (and working path MTU detection) could be useful, but with large (1500) byte packets and reasonable hardware, I never had trouble pushing 10G from the network side. Add TLS and some other processing, and older hardware wouldn't keep up, but not because of packetization. Small packets is also a different story. All my hardware at the time was xeon 2690, v1-4. Nics were Intel x520/x540 or similar (whatever SuperMicro was using back then). IIRC, v1 could do 10G easy without TLS, 8-9G with TLS, v3 improved AES acceleration and we could push 2x10G. When I turned off NIC packetization acceleration, I didn't notice much change in CPU or throughput, but if packetization was a bottleneck it should have been significant. At home, with similar age desktop processors with @ dual core pentium g3470 (haswell, same gen as a 2690v3), I can't quite hit 10g in iperf, but it's closeish, another two cores would probably do it. In some cases, you can get some big gains in efficiency by lining up the user space cpu with the kernel cpu that handles the rx/tx queues that the NIC hashes the connection to, though.

dale_glass 2y ago
I discovered that putting a 10G interface into a bridge implies a very significant slowdown. Linux has to do stuff on the CPU to do the bridging, so that turns off a large part of the card's acceleration. That's not a good thing for a server that runs a bunch of VMs. Fortunately SR-IOV exists, but it seems a tad silly to me that I have to do all this weird PCIe passthrough stuff just for this. It's nice, don't get me wrong, but a bit too exotic for what should be a simple setup.
- toast0 2y ago
  Oh yeah, bridged mode on my little pentium system brings perf way down. It was fine on 1G, but when I upgraded to 10G and wanted to hit numbers, I needed to stop doing software bridging. For me, I have slots and NICs, so I moved away from virtual ethernet on a bridge to actual ports; main host gets the 10G, and everything else gets to use 1G ports. No SR-IOV on my board, but that's ok.
- kjellsbells 2y ago
  I always found it ironic that virtualization made me have to care about the hardware more than I ever had to before. 2013 me: slap the app on a Dell. It'll be fine. 2017 me: aw crap, the nic doesnt support SR-IOV. What do you mean, I need a special driver? Oh lordy, I'm pinning a whole damn CPU just so DPDK can pull packets off the wire?

by toast0 2y ago

dale_glass 2y ago

I discovered that putting a 10G interface into a bridge implies a very significant slowdown. Linux has to do stuff on the CPU to do the bridging, so that turns off a large part of the card's acceleration. That's not a good thing for a server that runs a bunch of VMs. Fortunately SR-IOV exists, but it seems a tad silly to me that I have to do all this weird PCIe passthrough stuff just for this. It's nice, don't get me wrong, but a bit too exotic for what should be a simple setup.

toast0 2y ago
Oh yeah, bridged mode on my little pentium system brings perf way down. It was fine on 1G, but when I upgraded to 10G and wanted to hit numbers, I needed to stop doing software bridging. For me, I have slots and NICs, so I moved away from virtual ethernet on a bridge to actual ports; main host gets the 10G, and everything else gets to use 1G ports. No SR-IOV on my board, but that's ok.
kjellsbells 2y ago
I always found it ironic that virtualization made me have to care about the hardware more than I ever had to before. 2013 me: slap the app on a Dell. It'll be fine. 2017 me: aw crap, the nic doesnt support SR-IOV. What do you mean, I need a special driver? Oh lordy, I'm pinning a whole damn CPU just so DPDK can pull packets off the wire?