4 ms·
Its a single lane pcie gen2 interface. The max theoretical is 500MB/sec. So you can't ever touch 10G with it. In reality getting 75% of theoretical on PCIe ten
by StillBored 6y ago
Its a single lane pcie gen2 interface. The max theoretical is 500MB/sec. So you can't ever touch 10G with it. In reality getting 75% of theoretical on PCIe tends to be a rough upper limit on most PCIe interfaces, so the 3Gbit your seeing is pretty close to what one would expect.
edit: Oh its 3Gbit across 5 interfaces, one of which isn't PCIe, so the PCIe side is probably only running at about 50%. It might be interesting to see if the CPUs are pegged (or just one of them). Even so, PCIe on the rpi isn't coherent so that is going to slow things down too.
- geerlingguy 6y agoIt looks like the problem is `ksoftirqd` gets pegged at 100% and the system just queues up packets, slowing everything down. See: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/3#issuecomment-719781909 https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...
- StillBored 6y agoSo, this is sorta indicative of a RSS problem, but on the rpi it could be caused by other things. Check /proc/interrupts to assure you have balanced MSI's, although that itself could be a problem too. edit: run `perf top` to see if that gives you a better idea.
- geerlingguy 6y agoResults: 15.96% [kernel] [k] _raw_spin_unlock_irqrestore 12.81% [kernel] [k] mmiocpy 6.26% [kernel] [k] __copy_to_user_memcpy 6.02% [kernel] [k] __local_bh_enable_ip 5.13% [igb] [k] igb_poll When it hit full blast, I started getting "Events are being lost, check IO/CPU overload!"
- SoapSeller 6y agoAnother idea will be to increase interrupt coalescing via ethtool -c/C
- dualboot 6y agoThis is common even on x86 systems. You have to set the irq affinity to utilize the available CPU cores. There is a script included with the source you used to compile drivers called "set_irq_affinity" Ex (Sets IRQ Affinity for all available cores) : [path-to-i40epackage]/scripts/set_irq_affinity -x all ethX
- geerlingguy 6y agoSo like https://pastebin.com/2Z4UECPq https://pastebin.com/2Z4UECPq ? — this didn't make a difference in the overall performance :(
- dualboot 6y agoLooks like the script needs to be adjusted to function on the Pi. I wish I had the cycles and the kit on hand to play with this!
- InvaderFizz 6y agoI would suggest you go ahead and try jumbo frames[0] as that will significantly decrease the CPU load and overhead. I would also suggest using taskset[1] on each iperf server process to bind them each to a different cpu core. Finally, I would suggest UDP on iperf and let the sending Pi's just completely saturate the link. If you do all that, I think you have a good chance at achieving 3.5Gbps over just the Intel card. 0: https://blah.cloud/hardware/test-jumbo-frames-working/ https://blah.cloud/hardware/test-jumbo-frames-working/ 1: https://linux.die.net/man/1/taskset https://linux.die.net/man/1/taskset
- leptons 6y ago>It might be interesting to see if the CPUs are pegged (or just one of them). This is very likely the answer. I see a lot of people who think of the Pi as some kind of workhorse and are trying to use it for things that it simply can't do. The Pi is a great little piece of hardware, but it's not really made for this kind of thing. I'd never think about using a Raspberry Pi if I had to think about "saturating a NIC".
- geerlingguy 6y agoWell it can saturate up to two, and almost three, gigabit NICs now. So not too shabby. But I like to know the limits so I can plan out a project and know whether I'm safe using a Pi, or a 3-5x more expensive board or small PC :)