3 ms·

It looks like the problem is `ksoftirqd` gets pegged at 100% and the system just queues up packets, slowing everything down. See: https://github.com/geerlingguy

It looks like the problem is `ksoftirqd` gets pegged at 100% and the system just queues up packets, slowing everything down. See: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/3#issuecomment-719781909 https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

StillBored 6y ago
So, this is sorta indicative of a RSS problem, but on the rpi it could be caused by other things. Check /proc/interrupts to assure you have balanced MSI's, although that itself could be a problem too. edit: run `perf top` to see if that gives you a better idea.
- geerlingguy 6y ago
  Results: 15.96% [kernel] [k] _raw_spin_unlock_irqrestore 12.81% [kernel] [k] mmiocpy 6.26% [kernel] [k] __copy_to_user_memcpy 6.02% [kernel] [k] __local_bh_enable_ip 5.13% [igb] [k] igb_poll When it hit full blast, I started getting "Events are being lost, check IO/CPU overload!"
- SoapSeller 6y ago
  Another idea will be to increase interrupt coalescing via ethtool -c/C
dualboot 6y ago
This is common even on x86 systems. You have to set the irq affinity to utilize the available CPU cores. There is a script included with the source you used to compile drivers called "set_irq_affinity" Ex (Sets IRQ Affinity for all available cores) : [path-to-i40epackage]/scripts/set_irq_affinity -x all ethX
- geerlingguy 6y ago
  So like https://pastebin.com/2Z4UECPq https://pastebin.com/2Z4UECPq ? — this didn't make a difference in the overall performance :(
  - dualboot 6y ago
    Looks like the script needs to be adjusted to function on the Pi. I wish I had the cycles and the kit on hand to play with this!
InvaderFizz 6y ago
I would suggest you go ahead and try jumbo frames[0] as that will significantly decrease the CPU load and overhead. I would also suggest using taskset[1] on each iperf server process to bind them each to a different cpu core. Finally, I would suggest UDP on iperf and let the sending Pi's just completely saturate the link. If you do all that, I think you have a good chance at achieving 3.5Gbps over just the Intel card. 0: https://blah.cloud/hardware/test-jumbo-frames-working/ https://blah.cloud/hardware/test-jumbo-frames-working/ 1: https://linux.die.net/man/1/taskset https://linux.die.net/man/1/taskset

StillBored 6y ago

So, this is sorta indicative of a RSS problem, but on the rpi it could be caused by other things. Check /proc/interrupts to assure you have balanced MSI's, although that itself could be a problem too. edit: run `perf top` to see if that gives you a better idea.

geerlingguy 6y ago
Results: 15.96% [kernel] [k] _raw_spin_unlock_irqrestore 12.81% [kernel] [k] mmiocpy 6.26% [kernel] [k] __copy_to_user_memcpy 6.02% [kernel] [k] __local_bh_enable_ip 5.13% [igb] [k] igb_poll When it hit full blast, I started getting "Events are being lost, check IO/CPU overload!"
SoapSeller 6y ago
Another idea will be to increase interrupt coalescing via ethtool -c/C

dualboot 6y ago

This is common even on x86 systems. You have to set the irq affinity to utilize the available CPU cores. There is a script included with the source you used to compile drivers called "set_irq_affinity" Ex (Sets IRQ Affinity for all available cores) : [path-to-i40epackage]/scripts/set_irq_affinity -x all ethX

geerlingguy 6y ago
So like https://pastebin.com/2Z4UECPq https://pastebin.com/2Z4UECPq ? — this didn't make a difference in the overall performance :(
- dualboot 6y ago
  Looks like the script needs to be adjusted to function on the Pi. I wish I had the cycles and the kit on hand to play with this!

InvaderFizz 6y ago

I would suggest you go ahead and try jumbo frames[0] as that will significantly decrease the CPU load and overhead. I would also suggest using taskset[1] on each iperf server process to bind them each to a different cpu core. Finally, I would suggest UDP on iperf and let the sending Pi's just completely saturate the link. If you do all that, I think you have a good chance at achieving 3.5Gbps over just the Intel card. 0: https://blah.cloud/hardware/test-jumbo-frames-working/ https://blah.cloud/hardware/test-jumbo-frames-working/ 1: https://linux.die.net/man/1/taskset https://linux.die.net/man/1/taskset