2 ms·

I think I've found the bottleneck now that I have the setup up and running again today—ksoftirqd quickly hits 100% CPU and stays that way until the benchmark ru

I think I've found the bottleneck now that I have the setup up and running again today—ksoftirqd quickly hits 100% CPU and stays that way until the benchmark run completes. See: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/3#issuecomment-719781909 https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

syoc 6y ago
I would have a look at sending data with either DPDK (https://doc.dpdk.org/burst-replay/introduction.html https://doc.dpdk.org/burst-replay/introduction.html) or AF_PACKET and mmap (https://sites.google.com/site/packetmmap/ https://sites.google.com/site/packetmmap/ ) You can also use ethtool -C on the NICs on both ends of the connection to rate limit the irq signal handeling allowing you to optimize for throughput instead of latency.
iscfrc 6y ago
You might want to try enabling jumbo frames by setting the MTU to something >1500 bytes. Doing so should reduce the number of IRQs per unit of time since each frame will be carrying more data and therefore there will be fewer of them. According to the Intel 82580EB datasheet[1] it supports an MTU of "9.5KB." It's unclear if that means 9500 or 9728 bytes. I looked briefly for a datasheet that includes the ethernet specs. of the Broadcom 2711 but didn't immediately find anything. Recent versions of iproute2 can output the maximum MTU of an interface via: # Look for "maxmtu" in the output ip -d link list Barring that you can try incrementally upping the MTU until you run in to errors. The MTU of an interface can be set via: ip link set $interface mtu $mtu Note that for symmetrical testing via direct crossover you'll want to have the MTU be the same on each interface pair. [1] https://www.intel.com/content/www/us/en/embedded/products/networking/82580-eb-db-gbe-controller-datasheet.html https://www.intel.com/content/www/us/en/embedded/products/ne... (pg. 25, "Size of jumbo frames supported")
- geerlingguy 6y ago
  I set the MTU to its max (just over 9000 on the intel, heh), but that didn't make a difference. The one thing that did move the needle was overclocking the CPU to 2.147 GHz (from base 1.5 GHz clock), and that got me to 3.4 Gbps. So it seems to be a CPU constraint at this point.
  - neurostimulant 6y ago
    I wonder if using user-space tcp stack (or anything that could bypass the kernel) could push the number higher.
  - zamadatix 6y ago
    Did you change the mtu of the other sides as well? If not tcp will negotiate an mss that makes the larger mtu go unused.
    - geerlingguy 6y ago
      Oh shoot, completely forgot to do that, as I was testing a few things one after the other and it slipped my mind. I'll have to try and see if I can get a little more out.

syoc 6y ago

I would have a look at sending data with either DPDK (https://doc.dpdk.org/burst-replay/introduction.html https://doc.dpdk.org/burst-replay/introduction.html) or AF_PACKET and mmap (https://sites.google.com/site/packetmmap/ https://sites.google.com/site/packetmmap/ ) You can also use ethtool -C on the NICs on both ends of the connection to rate limit the irq signal handeling allowing you to optimize for throughput instead of latency.

iscfrc 6y ago

You might want to try enabling jumbo frames by setting the MTU to something >1500 bytes. Doing so should reduce the number of IRQs per unit of time since each frame will be carrying more data and therefore there will be fewer of them. According to the Intel 82580EB datasheet[1] it supports an MTU of "9.5KB." It's unclear if that means 9500 or 9728 bytes. I looked briefly for a datasheet that includes the ethernet specs. of the Broadcom 2711 but didn't immediately find anything. Recent versions of iproute2 can output the maximum MTU of an interface via: # Look for "maxmtu" in the output ip -d link list Barring that you can try incrementally upping the MTU until you run in to errors. The MTU of an interface can be set via: ip link set $interface mtu $mtu Note that for symmetrical testing via direct crossover you'll want to have the MTU be the same on each interface pair. [1] https://www.intel.com/content/www/us/en/embedded/products/networking/82580-eb-db-gbe-controller-datasheet.html https://www.intel.com/content/www/us/en/embedded/products/ne... (pg. 25, "Size of jumbo frames supported")

geerlingguy 6y ago
I set the MTU to its max (just over 9000 on the intel, heh), but that didn't make a difference. The one thing that did move the needle was overclocking the CPU to 2.147 GHz (from base 1.5 GHz clock), and that got me to 3.4 Gbps. So it seems to be a CPU constraint at this point.
- neurostimulant 6y ago
  I wonder if using user-space tcp stack (or anything that could bypass the kernel) could push the number higher.
- zamadatix 6y ago
  Did you change the mtu of the other sides as well? If not tcp will negotiate an mss that makes the larger mtu go unused.
  - geerlingguy 6y ago
    Oh shoot, completely forgot to do that, as I was testing a few things one after the other and it slipped my mind. I'll have to try and see if I can get a little more out.