3 ms·

Sorry about the slightly-clickbaity title. I actually have at least a 10 GbE card (and switch) on the way to test those and see if I can get more out of it, but

Sorry about the slightly-clickbaity title. I actually have at least a 10 GbE card (and switch) on the way to test those and see if I can get more out of it, but for _this_ test, I had a 4-interface Intel I340-T4, and I managed to get a maximum throughput of 3.06 Gbps when pumping bits through all 4 of those plus the built-in Gigabit interface on the Compute Module. For some reason I couldn't break that barrier, even though all the interfaces can do ~940 Mbps on their own, and any three on the PCIe card can do ~2.8 Gbps. It seems like there's some sort of upper limit around 3 Gbps on the Pi CM4 (even when combining the internal interface) :-/ But maybe I'm missing something in the Pi OS / Debian/Linux kernel stack that is holding me back? Or is it a limitation on the SoC? I though the ethernet chip was separate from the PCIe lanes on it, but maybe there's something internal to the BCM2711 that's bottlenecking it. Also... tons more detail here: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/3 https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

monocasa 6y ago
You might be hitting the limits of the RAM. I think LPDDR3 maxes out at ~4.2Gbps, and running other bus masters like the HDMI and OS itself would be cutting into that.
- wmf 6y ago
  32-bit LPDDR4-3200 should give 12.8 Gbytes/s which is 102 Gbits/s.
  - monocasa 6y ago
    You can't just multiply width*frequency for DRAM these days, as much as I wish we still lived in the days of ubiquitous SRAM. The chip in some of the 2GB RPI4s is rated for only 3.7Gbps. https://www.samsung.com/semiconductor/dram/lpddr4/K4F6E304HB-MGCJ/ https://www.samsung.com/semiconductor/dram/lpddr4/K4F6E304HB...
    - mmastrac 6y ago
      Is there a way to see if you are hitting memory bandwidth issues in Linux?
      - monocasa 6y ago
        Not in a holistic way AFAIK, and for sure not rigged up to the Raspbian kernel (since all of that lives on the videocore side), but I bet Broadcom or the RPi foundation has access to some undocumented perf counters on the DRAM controller that could illuminate this if they were the ones debugging it.
    - wmf 6y ago
      No, that chip is rated for 3.7 Gbps per pin and it's 32 bits wide. Even at ~60% efficiency you're an order of magnitude off.
      - monocasa 6y ago
        Real world tests are seeing around 3 to 4 Gbps of memory bandwidth. https://medium.com/@ghalfacree/benchmarking-the-raspberry-pi-4-73e5afbcd54b https://medium.com/@ghalfacree/benchmarking-the-raspberry-pi... LPDDR cannot sustain anywhere near the max speed of the interface. It's more of a hope that you can burst something out and go to sleep rather than trying to maintain that speed. In a lot of ways DRAM hasn't gotten faster in decades when you look at how latency clocks nearly always increase at the same rate of interface speed increases. And LPDDR is the niche where that shines the most, because it doesn't have oodles of dies to interleave to hide that issue.
        hedgehog 6y ago
        Those numbers look way off, maybe they mixed up the units? Should be a few GBps at least.
        wmf 6y ago
        Innumeracy strikes again. It's actually 4-5 Gbytes/s [1] plus whatever bandwidth the video scanout is stealing (~400 Mbytes/s?). That's only ~40% efficient which is simultaneously terrible and pretty much what you'd expect from Broadcom. However 4 Gbytes/s is 32 Gbits/s which leaves plenty of headroom to do 5 Gbits/s of network I/O. [1] https://www.raspberrypi.org/forums/viewtopic.php?t=271121 https://www.raspberrypi.org/forums/viewtopic.php?t=271121
mmastrac 6y ago
Awesome work. Been watching your videos on these (the video card one was especially interesting). At what point are you saturating the poor little ARM CPU (or its tiny PCIe interface)?
- geerlingguy 6y ago
  Heh, I know that ~3 Gbps is the maximum you can get through the PCIe interface (x1, PCI 2.0), so that is expected. But I was hoping the internal ethernet interface was separate and could add one 1 Gbps more... the CPU didn't seem to be maxed out and was also not overheating at the time (especially not with my 12" fan blasting on it).
  - dualboot 6y ago
    with some tuning you should be able to saturate the PCIe 1x slot. Excellent reading on this available here : http://www.intel.com/content/dam/doc/application-note/82575-82576-82598-82599-ethernet-controllers-latency-appl-note.pdf http://www.intel.com/content/dam/doc/application-note/82575-... and here : https://blog.cloudflare.com/how-to-achieve-low-latency/ https://blog.cloudflare.com/how-to-achieve-low-latency/ Edit : with the inbound 10Gb card referenced
  - toast0 6y ago
    Was all this TCP? You might try UDP as well, in case you're hitting a bottleneck in the tcp stack.
wil421 6y ago
Do you think an SFP+ nic would work? It would be cool to try out fiber.
- baybal2 6y ago
  There are no SFP option on 5gbps NICs as i understand as per standard
ksec 6y ago
>Sorry about the slightly-clickbaity title. Well yes because 5Gbps Ethernet is actually a thing ( NBase-T or 5GBASE-T). So 1Gbps x 5 would be more accurate. Cant wait to see results on 10GbE though :) P.S I really wish 5Gbps Ethernet is more common.
- ncrmro 6y ago
  My ATT router made by Nokia has one 5gbe and the fiber plugs in directly with SFP!
- geerlingguy 6y ago
  True true... though in my work trying to get a flexible 10 GbE network set up in my house, I've found that the support for 2.5 and 5 GbE are iffy at best on many devices :(
  - voltagex_ 6y ago
    The "best" I've found so far (and gives you options to go 2.5GbE/5GbE 1: https://www.amazon.com/UGREEN-Ethernet-Thunderbolt-Converter-Compatible/dp/B081TY1WQX https://www.amazon.com/UGREEN-Ethernet-Thunderbolt-Converter... (USB-C) 2: https://www.amazon.com/2-5GBase-T-Ethernet-Controller-Standard-Low-Profile/dp/B07Y2GWVB8 https://www.amazon.com/2-5GBase-T-Ethernet-Controller-Standa... (Don't get the knock-off version of this, the brackets aren't the right sizes.) (PCIe) The expensive ones I'm waiting to arrive: 3: Either a second hand Intel X520-DA1 card or the "refurb" from AliExpress and https://mikrotik.com/product/crs305_1g_4s_in https://mikrotik.com/product/crs305_1g_4s_in with RJ10 SFP+ modules. Then cry at how much you just spent.
    - johnwalkr 6y ago
      X520-DA1 and most older stuff doesn’t support 2.5Gbps or 5Gbps (only 1 and 10).
      - voltagex_ 6y ago
        Oops. They haven't arrived yet. Good thing they weren't too expensive. What would be my best bet for supporting all of those speeds?
StillBored 6y ago
Its a single lane pcie gen2 interface. The max theoretical is 500MB/sec. So you can't ever touch 10G with it. In reality getting 75% of theoretical on PCIe tends to be a rough upper limit on most PCIe interfaces, so the 3Gbit your seeing is pretty close to what one would expect. edit: Oh its 3Gbit across 5 interfaces, one of which isn't PCIe, so the PCIe side is probably only running at about 50%. It might be interesting to see if the CPUs are pegged (or just one of them). Even so, PCIe on the rpi isn't coherent so that is going to slow things down too.
- geerlingguy 6y ago
  It looks like the problem is `ksoftirqd` gets pegged at 100% and the system just queues up packets, slowing everything down. See: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/3#issuecomment-719781909 https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...
  - StillBored 6y ago
    So, this is sorta indicative of a RSS problem, but on the rpi it could be caused by other things. Check /proc/interrupts to assure you have balanced MSI's, although that itself could be a problem too. edit: run `perf top` to see if that gives you a better idea.
    - geerlingguy 6y ago
      Results: 15.96% [kernel] [k] _raw_spin_unlock_irqrestore 12.81% [kernel] [k] mmiocpy 6.26% [kernel] [k] __copy_to_user_memcpy 6.02% [kernel] [k] __local_bh_enable_ip 5.13% [igb] [k] igb_poll When it hit full blast, I started getting "Events are being lost, check IO/CPU overload!"
    - SoapSeller 6y ago
      Another idea will be to increase interrupt coalescing via ethtool -c/C
stratosmacker 6y ago
Jeff, First off, thank you for doing this kind of 'r&d', it is really exciting to see what the Pi is capable of after less than a decade. Would you be interested in someone testing a SAS PCI card? I'm going to pick up one of these as soon as they're not backordered...
CyberDildonics 6y ago
Instead of lying and then apologizing once you get what you want, it would be better to just not lie in the first place.
- geerlingguy 6y ago
  Technically it's not a lie—there are 5x1 Gbps of interfaces here. But I wanted to acknowledge that I used a technicality to get the title how I wanted it, because if I didn't do that, a lot of people wouldn't read it, and then we wouldn't get to have this enlightening discussion ;)
  - CyberDildonics 6y ago
    You could hook up a 100gbs card, but that wouldn't make it 100gbs ethernet on a raspberry pi.
    - IntelMiner 6y ago
      It would, but it wouldn't be able to push 100gbs. It's not lying
      - geerlingguy 6y ago
        I'm just looking around for this mystery 100 Gbps card. Think it'll work with Cat9?
        CyberDildonics 6y ago
        Did you think this was made up for some reason? 100Gb cards are not new. QSFP28 was created in 2014. https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p1100p https://www.broadcom.com/products/ethernet-connectivity/netw... They don't use copper, they use fiber. It wouldn't be a mystery if you searched for '100gbs pci ethernet'.
soneil 6y ago
I assume you saw the video with Plunkett the RPF put out ( https://youtu.be/yiHgmNBOzkc https://youtu.be/yiHgmNBOzkc specifically interesting at 10:45 ) - He mentioned he was testing 10GbE fibre and reached 3.2gbit. Now he goes into absolutely zero detail on that, but I find it interesting you've both hit the same ceiling. (He also mentioned 390MB/sec write speed to nvme, which is suspiciously close to the same ceiling)
- geerlingguy 6y ago
  Yeah, I think the PCIe link hits a ceiling around there. Note that combining the internal interface with the 4 NIC interfaces, and overclocking to 2.147 GHz got it up to 3.4 total Gbps. So the IRQ interrupts are the main bottleneck when it comes to total network packet throughout.
  - jlgaddis 6y ago
    Since you're also from the midwest, I'll put it in terms you'll understand: :-) > I think the PCIe link hits a ceiling around there. You're trying to shove 10 gallons of shit into a 5-gallon bucket! -- I'm not sure how high you can set the MTU on those Pi's (the Intels should handle 9000) but I'd set them as high as they'll go, if I were you. An MTU of 9000 basically means ~1/6th the interrupts.

monocasa 6y ago

You might be hitting the limits of the RAM. I think LPDDR3 maxes out at ~4.2Gbps, and running other bus masters like the HDMI and OS itself would be cutting into that.

wmf 6y ago
32-bit LPDDR4-3200 should give 12.8 Gbytes/s which is 102 Gbits/s.
- monocasa 6y ago
  You can't just multiply width*frequency for DRAM these days, as much as I wish we still lived in the days of ubiquitous SRAM. The chip in some of the 2GB RPI4s is rated for only 3.7Gbps. https://www.samsung.com/semiconductor/dram/lpddr4/K4F6E304HB-MGCJ/ https://www.samsung.com/semiconductor/dram/lpddr4/K4F6E304HB...
  - mmastrac 6y ago
    Is there a way to see if you are hitting memory bandwidth issues in Linux?
    - monocasa 6y ago
      Not in a holistic way AFAIK, and for sure not rigged up to the Raspbian kernel (since all of that lives on the videocore side), but I bet Broadcom or the RPi foundation has access to some undocumented perf counters on the DRAM controller that could illuminate this if they were the ones debugging it.
  - wmf 6y ago
    No, that chip is rated for 3.7 Gbps per pin and it's 32 bits wide. Even at ~60% efficiency you're an order of magnitude off.
    - monocasa 6y ago
      Real world tests are seeing around 3 to 4 Gbps of memory bandwidth. https://medium.com/@ghalfacree/benchmarking-the-raspberry-pi-4-73e5afbcd54b https://medium.com/@ghalfacree/benchmarking-the-raspberry-pi... LPDDR cannot sustain anywhere near the max speed of the interface. It's more of a hope that you can burst something out and go to sleep rather than trying to maintain that speed. In a lot of ways DRAM hasn't gotten faster in decades when you look at how latency clocks nearly always increase at the same rate of interface speed increases. And LPDDR is the niche where that shines the most, because it doesn't have oodles of dies to interleave to hide that issue.
      - hedgehog 6y ago
        Those numbers look way off, maybe they mixed up the units? Should be a few GBps at least.
      - wmf 6y ago
        Innumeracy strikes again. It's actually 4-5 Gbytes/s [1] plus whatever bandwidth the video scanout is stealing (~400 Mbytes/s?). That's only ~40% efficient which is simultaneously terrible and pretty much what you'd expect from Broadcom. However 4 Gbytes/s is 32 Gbits/s which leaves plenty of headroom to do 5 Gbits/s of network I/O. [1] https://www.raspberrypi.org/forums/viewtopic.php?t=271121 https://www.raspberrypi.org/forums/viewtopic.php?t=271121

mmastrac 6y ago

Awesome work. Been watching your videos on these (the video card one was especially interesting). At what point are you saturating the poor little ARM CPU (or its tiny PCIe interface)?

geerlingguy 6y ago
Heh, I know that ~3 Gbps is the maximum you can get through the PCIe interface (x1, PCI 2.0), so that is expected. But I was hoping the internal ethernet interface was separate and could add one 1 Gbps more... the CPU didn't seem to be maxed out and was also not overheating at the time (especially not with my 12" fan blasting on it).
- dualboot 6y ago
  with some tuning you should be able to saturate the PCIe 1x slot. Excellent reading on this available here : http://www.intel.com/content/dam/doc/application-note/82575-82576-82598-82599-ethernet-controllers-latency-appl-note.pdf http://www.intel.com/content/dam/doc/application-note/82575-... and here : https://blog.cloudflare.com/how-to-achieve-low-latency/ https://blog.cloudflare.com/how-to-achieve-low-latency/ Edit : with the inbound 10Gb card referenced
- toast0 6y ago
  Was all this TCP? You might try UDP as well, in case you're hitting a bottleneck in the tcp stack.

wil421 6y ago

Do you think an SFP+ nic would work? It would be cool to try out fiber.

baybal2 6y ago
There are no SFP option on 5gbps NICs as i understand as per standard

ksec 6y ago

>Sorry about the slightly-clickbaity title. Well yes because 5Gbps Ethernet is actually a thing ( NBase-T or 5GBASE-T). So 1Gbps x 5 would be more accurate. Cant wait to see results on 10GbE though :) P.S I really wish 5Gbps Ethernet is more common.

ncrmro 6y ago
My ATT router made by Nokia has one 5gbe and the fiber plugs in directly with SFP!
geerlingguy 6y ago
True true... though in my work trying to get a flexible 10 GbE network set up in my house, I've found that the support for 2.5 and 5 GbE are iffy at best on many devices :(
- voltagex_ 6y ago
  The "best" I've found so far (and gives you options to go 2.5GbE/5GbE 1: https://www.amazon.com/UGREEN-Ethernet-Thunderbolt-Converter-Compatible/dp/B081TY1WQX https://www.amazon.com/UGREEN-Ethernet-Thunderbolt-Converter... (USB-C) 2: https://www.amazon.com/2-5GBase-T-Ethernet-Controller-Standard-Low-Profile/dp/B07Y2GWVB8 https://www.amazon.com/2-5GBase-T-Ethernet-Controller-Standa... (Don't get the knock-off version of this, the brackets aren't the right sizes.) (PCIe) The expensive ones I'm waiting to arrive: 3: Either a second hand Intel X520-DA1 card or the "refurb" from AliExpress and https://mikrotik.com/product/crs305_1g_4s_in https://mikrotik.com/product/crs305_1g_4s_in with RJ10 SFP+ modules. Then cry at how much you just spent.
  - johnwalkr 6y ago
    X520-DA1 and most older stuff doesn’t support 2.5Gbps or 5Gbps (only 1 and 10).
    - voltagex_ 6y ago
      Oops. They haven't arrived yet. Good thing they weren't too expensive. What would be my best bet for supporting all of those speeds?

StillBored 6y ago

Its a single lane pcie gen2 interface. The max theoretical is 500MB/sec. So you can't ever touch 10G with it. In reality getting 75% of theoretical on PCIe tends to be a rough upper limit on most PCIe interfaces, so the 3Gbit your seeing is pretty close to what one would expect. edit: Oh its 3Gbit across 5 interfaces, one of which isn't PCIe, so the PCIe side is probably only running at about 50%. It might be interesting to see if the CPUs are pegged (or just one of them). Even so, PCIe on the rpi isn't coherent so that is going to slow things down too.

geerlingguy 6y ago
It looks like the problem is `ksoftirqd` gets pegged at 100% and the system just queues up packets, slowing everything down. See: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/3#issuecomment-719781909 https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...
- StillBored 6y ago
  So, this is sorta indicative of a RSS problem, but on the rpi it could be caused by other things. Check /proc/interrupts to assure you have balanced MSI's, although that itself could be a problem too. edit: run `perf top` to see if that gives you a better idea.
  - geerlingguy 6y ago
    Results: 15.96% [kernel] [k] _raw_spin_unlock_irqrestore 12.81% [kernel] [k] mmiocpy 6.26% [kernel] [k] __copy_to_user_memcpy 6.02% [kernel] [k] __local_bh_enable_ip 5.13% [igb] [k] igb_poll When it hit full blast, I started getting "Events are being lost, check IO/CPU overload!"
  - SoapSeller 6y ago
    Another idea will be to increase interrupt coalescing via ethtool -c/C

stratosmacker 6y ago

Jeff, First off, thank you for doing this kind of 'r&d', it is really exciting to see what the Pi is capable of after less than a decade. Would you be interested in someone testing a SAS PCI card? I'm going to pick up one of these as soon as they're not backordered...

CyberDildonics 6y ago

Instead of lying and then apologizing once you get what you want, it would be better to just not lie in the first place.

geerlingguy 6y ago
Technically it's not a lie—there are 5x1 Gbps of interfaces here. But I wanted to acknowledge that I used a technicality to get the title how I wanted it, because if I didn't do that, a lot of people wouldn't read it, and then we wouldn't get to have this enlightening discussion ;)
- CyberDildonics 6y ago
  You could hook up a 100gbs card, but that wouldn't make it 100gbs ethernet on a raspberry pi.
  - IntelMiner 6y ago
    It would, but it wouldn't be able to push 100gbs. It's not lying
    - geerlingguy 6y ago
      I'm just looking around for this mystery 100 Gbps card. Think it'll work with Cat9?
      - CyberDildonics 6y ago
        Did you think this was made up for some reason? 100Gb cards are not new. QSFP28 was created in 2014. https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p1100p https://www.broadcom.com/products/ethernet-connectivity/netw... They don't use copper, they use fiber. It wouldn't be a mystery if you searched for '100gbs pci ethernet'.

soneil 6y ago

I assume you saw the video with Plunkett the RPF put out ( https://youtu.be/yiHgmNBOzkc https://youtu.be/yiHgmNBOzkc specifically interesting at 10:45 ) - He mentioned he was testing 10GbE fibre and reached 3.2gbit. Now he goes into absolutely zero detail on that, but I find it interesting you've both hit the same ceiling. (He also mentioned 390MB/sec write speed to nvme, which is suspiciously close to the same ceiling)

geerlingguy 6y ago
Yeah, I think the PCIe link hits a ceiling around there. Note that combining the internal interface with the 4 NIC interfaces, and overclocking to 2.147 GHz got it up to 3.4 total Gbps. So the IRQ interrupts are the main bottleneck when it comes to total network packet throughout.
- jlgaddis 6y ago
  Since you're also from the midwest, I'll put it in terms you'll understand: :-) > I think the PCIe link hits a ceiling around there. You're trying to shove 10 gallons of shit into a 5-gallon bucket! -- I'm not sure how high you can set the MTU on those Pi's (the Intels should handle 9000) but I'd set them as high as they'll go, if I were you. An MTU of 9000 basically means ~1/6th the interrupts.