4 ms·

So theoretically, 5 Gbps was possible No, it is not. That NIC is a PCIe Gen2 NIC. By using only a single lane, you're limiting the bandwidth to ~500MB/sec th

So theoretically, 5 Gbps was possible No, it is not. That NIC is a PCIe Gen2 NIC. By using only a single lane, you're limiting the bandwidth to ~500MB/sec theoretical. That's 4Gb/s theoretical, and getting 3Gb/s is ~75% of the theoretical bandwidth, which is pretty decent.

geerlingguy 6y ago
I'll take pretty decent, then :) I mean, before this the most I had tested successfully was a little over 2 Gbps with three NICs on a Pi 4 B.
- drewg123 6y ago
  Can you run an lspci -vvv on the Intel NIC? I just re-read things, and it seems like 1 of those Gb/s is coming from the on-board NIC. I'm curious if maybe PCIe is running at Gen1
  - geerlingguy 6y ago
    Here you go! https://pastebin.com/A8gsGz3t https://pastebin.com/A8gsGz3t
    - drewg123 6y ago
      So its running Gen2 x1, which is good. I was afraid that it might have downshifted to Gen1. Other threads point to your CPU being pegged, and I would tend to agree with that. What direction are you running the streams in? In general, sending is much more efficient than receiving ("its better to give than to receive"). From your statement that ksoftirqd is pegged, I'm guessing you're receiving. I'd first see what bandwidth you can send at with iperf when you run the test in reverse so this pi is sending. Then, to eliminate memory bw as a potential bottleneck, you could use sendfile. I don't think iperf ever supported sendfile (but its been years since I've used it). I'd suggest installing netperf on this pi, running netserver on its link partners, and running "netperf -tTCP_SENDFILE -H othermachine" to all 5 peers and see what happens.
stkdump 6y ago
Well, when a LAN is 1Gb/s they are actually not talking about real bits. It actually is 100MB/s max, not 125MB/s as one might expect. Back in the old days they used to call it baud.
- wmf 6y ago
  This is wrong; 1 Gbps Ethernet is 125 MB/s (including headers/trailer and inter-packet gap so you only get ~117 in practice). Infiniband, SATA, and Fibre Channel cheat but Ethernet doesn't.
  - hinkley 6y ago
    The 10:1 bits/bytes ratio common in some kinds of equipment is in fact a 5:4 encoding to make it easier to detect bit boundaries and to avoid various electrical problems with the signal. Modems used to do this too. The 'cheat' is that they report Layer 1 bandwidth, which is a completely useless number to the end user. The bulk of the loss occurs between Layer 1 and Layer 2 (with dribs and drabs for packet headers and so forth)

geerlingguy 6y ago

I'll take pretty decent, then :) I mean, before this the most I had tested successfully was a little over 2 Gbps with three NICs on a Pi 4 B.

drewg123 6y ago
Can you run an lspci -vvv on the Intel NIC? I just re-read things, and it seems like 1 of those Gb/s is coming from the on-board NIC. I'm curious if maybe PCIe is running at Gen1
- geerlingguy 6y ago
  Here you go! https://pastebin.com/A8gsGz3t https://pastebin.com/A8gsGz3t
  - drewg123 6y ago
    So its running Gen2 x1, which is good. I was afraid that it might have downshifted to Gen1. Other threads point to your CPU being pegged, and I would tend to agree with that. What direction are you running the streams in? In general, sending is much more efficient than receiving ("its better to give than to receive"). From your statement that ksoftirqd is pegged, I'm guessing you're receiving. I'd first see what bandwidth you can send at with iperf when you run the test in reverse so this pi is sending. Then, to eliminate memory bw as a potential bottleneck, you could use sendfile. I don't think iperf ever supported sendfile (but its been years since I've used it). I'd suggest installing netperf on this pi, running netserver on its link partners, and running "netperf -tTCP_SENDFILE -H othermachine" to all 5 peers and see what happens.

stkdump 6y ago

Well, when a LAN is 1Gb/s they are actually not talking about real bits. It actually is 100MB/s max, not 125MB/s as one might expect. Back in the old days they used to call it baud.

wmf 6y ago
This is wrong; 1 Gbps Ethernet is 125 MB/s (including headers/trailer and inter-packet gap so you only get ~117 in practice). Infiniband, SATA, and Fibre Channel cheat but Ethernet doesn't.
- hinkley 6y ago
  The 10:1 bits/bytes ratio common in some kinds of equipment is in fact a 5:4 encoding to make it easier to detect bit boundaries and to avoid various electrical problems with the signal. Modems used to do this too. The 'cheat' is that they report Layer 1 bandwidth, which is a completely useless number to the end user. The bulk of the loss occurs between Layer 1 and Layer 2 (with dribs and drabs for packet headers and so forth)