4 ms·

Most of the time 10G routing isn’t for WAN but instead for the LAN.

PragmaticPulp 6y ago
I think everyone is missing the point of this cheap hobbyist board. The PCIe slot on the H2 is PCIe 2.0 x4, which doesn't have enough bandwidth to saturate more than one 10G port. This is why hobby hardware and the enthusiast market is so difficult: You can make a great product at a cheap price point and people will still complain that it doesn't satisfy their high-end niche edge case.
- Plutoberth 6y ago
  This board has four 2.5G ports, not four 10G ports, so the 16G bandwidth for four PCie 2.0 lanes is more than enough.
  - mlyle 6y ago
    Right... the parent is saying that 4x10G is pointless compared to 4x2.5G, because PCIe 4 lanes will top out at forwarding around 7gbps of traffic. You can't do line rate on all ports either (limited by PCIe alone, let alone CPU for smaller packets), but you can certainly fill an individual port, which I suspect is the goal.
    - q3k 6y ago
      > because PCIe 4 lanes will top out at forwarding around 7gbps of traffic [...] limited by PCIe alone Are you sure about that? With 5GT/s (or 500MB/s) per lane, and with 4 lanes, that should be plenty, no? Intel adapters like the x520-DA2 are specced at 2x 10G, and use PCIe 2.0 x8. FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.
      - mlyle 6y ago
        > FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0. Yah, I said forwarding-- so each packet goes over the PCIe link twice. So, to use your test-- 3.7 x 4 / 2 = 7.4gbps. I said around 7. 7.4gbps would be a bit more than I'd expect, but I'd not be shocked.
        q3k 6y ago
        But PCIe is full duplex! With PCIe 2.0 x4 there's 4 lanes in each direction [1], so when 'forwarding' over a single 10G link you can expect to send and receive simultaneously at the speed I mentioned earlier. [1] - https://en.wikipedia.org/wiki/PCI_Express#Pinout https://en.wikipedia.org/wiki/PCI_Express#Pinout
        mlyle 6y ago
        Yah, I guess dividing by 2 isn't fair. But transmitting does impact receiving and vice-versa: when you're reading DMA descriptors, you need to wait/hold for posted completions, etc. It's not fully uncontended between send and receive, but more uncontended than a naive division by 2 would imply.
      - adrian_b 6y ago
        The so-called 5 Gb/s PCIe 2.0 lanes are in fact 4 Gb/s lanes. Ethernet speeds are given in data bits per second, while USB 3.0, PCIe 1.0 and PCIe 2.0 add in their speed values the extra bits used for encoding. On the other hand, 10 Gb/s USB and 8 Gb/s PCIe 3.0 have really those speeds. Complicated :-( One PCIe 2.0 lane is more than enough for 2.5 Gb/s Ethernet, but not enough for 5 Gb/s Ethernet or 10 Gb/s Ethernet.
- adrian_b 6y ago
  You are right that 4 PCIe 2.0 lanes have an aggregate speed of 16 Gb/s (even if in marketing speak they would be wrongly called as 20 Gb/s), which is not enough even for two 10 Gb/s ports, but only for one. However, that is not due to being a cheap hobbyist board, but due to Intel, who have not provided their Atom line of CPUs with more lanes or with faster lanes. Only the Elkhart Lake and Jasper Lake CPUs, which have been just launched by Intel as a replacement for the Gemini Lake Refresh CPUs used in ODROID H2, are the first Intel CPUs of the Atom series that have PCIe 3.0.

I think everyone is missing the point of this cheap hobbyist board. The PCIe slot on the H2 is PCIe 2.0 x4, which doesn't have enough bandwidth to saturate more than one 10G port. This is why hobby hardware and the enthusiast market is so difficult: You can make a great product at a cheap price point and people will still complain that it doesn't satisfy their high-end niche edge case.

Plutoberth 6y ago
This board has four 2.5G ports, not four 10G ports, so the 16G bandwidth for four PCie 2.0 lanes is more than enough.
- mlyle 6y ago
  Right... the parent is saying that 4x10G is pointless compared to 4x2.5G, because PCIe 4 lanes will top out at forwarding around 7gbps of traffic. You can't do line rate on all ports either (limited by PCIe alone, let alone CPU for smaller packets), but you can certainly fill an individual port, which I suspect is the goal.
  - q3k 6y ago
    > because PCIe 4 lanes will top out at forwarding around 7gbps of traffic [...] limited by PCIe alone Are you sure about that? With 5GT/s (or 500MB/s) per lane, and with 4 lanes, that should be plenty, no? Intel adapters like the x520-DA2 are specced at 2x 10G, and use PCIe 2.0 x8. FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.
    - mlyle 6y ago
      > FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0. Yah, I said forwarding-- so each packet goes over the PCIe link twice. So, to use your test-- 3.7 x 4 / 2 = 7.4gbps. I said around 7. 7.4gbps would be a bit more than I'd expect, but I'd not be shocked.
      - q3k 6y ago
        But PCIe is full duplex! With PCIe 2.0 x4 there's 4 lanes in each direction [1], so when 'forwarding' over a single 10G link you can expect to send and receive simultaneously at the speed I mentioned earlier. [1] - https://en.wikipedia.org/wiki/PCI_Express#Pinout https://en.wikipedia.org/wiki/PCI_Express#Pinout
        mlyle 6y ago
        Yah, I guess dividing by 2 isn't fair. But transmitting does impact receiving and vice-versa: when you're reading DMA descriptors, you need to wait/hold for posted completions, etc. It's not fully uncontended between send and receive, but more uncontended than a naive division by 2 would imply.
    - adrian_b 6y ago
      The so-called 5 Gb/s PCIe 2.0 lanes are in fact 4 Gb/s lanes. Ethernet speeds are given in data bits per second, while USB 3.0, PCIe 1.0 and PCIe 2.0 add in their speed values the extra bits used for encoding. On the other hand, 10 Gb/s USB and 8 Gb/s PCIe 3.0 have really those speeds. Complicated :-( One PCIe 2.0 lane is more than enough for 2.5 Gb/s Ethernet, but not enough for 5 Gb/s Ethernet or 10 Gb/s Ethernet.
adrian_b 6y ago
You are right that 4 PCIe 2.0 lanes have an aggregate speed of 16 Gb/s (even if in marketing speak they would be wrongly called as 20 Gb/s), which is not enough even for two 10 Gb/s ports, but only for one. However, that is not due to being a cheap hobbyist board, but due to Intel, who have not provided their Atom line of CPUs with more lanes or with faster lanes. Only the Elkhart Lake and Jasper Lake CPUs, which have been just launched by Intel as a replacement for the Gemini Lake Refresh CPUs used in ODROID H2, are the first Intel CPUs of the Atom series that have PCIe 3.0.