4 ms·

I planned to use the ZimaBoard as a small OpnSense platform. Sadly, the NICs are Broadcom which has an unhealthy tendency to fail in weird way when throughput i

I planned to use the ZimaBoard as a small OpnSense platform. Sadly, the NICs are Broadcom which has an unhealthy tendency to fail in weird way when throughput is pushed (lucky enough that I have fiber Intarwebz, so Gig is not unreasonable). I am not alone, so it is not one-off. [1] The solution is use an Intel NIC off the PCIe slot exposed by ZimaBoard, but there isn't a decent bracket to hold it. There is a 3D printable design [2] (unaffiliated), but ehhh, as been pointed out, used small business PCs are plentiful. I went with a used Sophos SG115 rev.2 which uses Intel Pro/1000 NICs, which works splendidly. [1] https://www.reddit.com/r/ZimaBoard/comments/12gij2d/zimaboard_opnsense_wan_port_dropping/ https://www.reddit.com/r/ZimaBoard/comments/12gij2d/zimaboar... [2] https://www.printables.com/model/215289-zimaboard-wall-mount-for-generic-pcie-cards https://www.printables.com/model/215289-zimaboard-wall-mount...

Fischgericht 3y ago
Agreed, I'd never ever buy a product that has a Broadcom NIC or any other networking equipment based on Broadcom NICs. As an example, their 10G BCM53161 controllers have a power consumption of TWICE of the competing Realteks RTL8372. However, the Zimaboard specs claim to be using Realtek 8111H. Maybe they changed the design recently? Now, the 8111H also has a hell lot of issues, again recently with Linux Kernel 6.2, but it's nowhere as bad as the Broadcom stuff. For maybe $2 more they could have used some PROVEN (aka old) Intel NIC instead. (The recent Intel 2.5G I225-VI and 226-V ALSO have a bunch of problems/bugs, sigh) In general, quality of NIC chips has gone down during the last 10 years. If you want something to be used for critical/stable networking, you really have to do some research.
- bcrl 3y ago
  Intel NICs have had similar problems. One of my circa 2010 servers had the NIC lock up in such a way that passthru traffic to the IPMI was blocked preventing remote management access. I drove 3.5 hours to the data center to power cycle the system. Investigation eventually turned up an errata for exactly that issue. Suffice it to say that I decommissioned that server as soon as I could.
  - T3OU-736 3y ago
    > Intel NICs have had similar problems. One of my circa 2010 servers had the NIC lock up in such a way that passthru traffic to the IPMI was blocked preventing remote management access. I drove 3.5 hours to the data center to power cycle the system. Investigation eventually turned up an errata for exactly that issue. Suffice it to say that I decommissioned that server as soon as I could. In the past life of being a SysAdmin, we strove for independent OOB NICs whenever possible specifically for this sort of edge case. And then used switched type PDUs (instead of just metered) so that remote power-cycling was possible for the eventuality that the IPMI controller had an OOM and no incantation of `ipmitool` would suffice (<caugh> iDRAC <caugh>). Sounds like you weren't able to get all those nice things. Unfortunate. Been there.
  - Fischgericht 3y ago
    In-Band passthru is a nightmare. Even if your NIC is good, it can cause all sorts of problems. It's okayish if it's done via an Ethernet bridge, but that seems to be the exception. On the ASRock Rack boards I have recently sourced you suddenly get a new PCIe root bridge, and (10G) NIC throughput goes down the drain.
- T3OU-736 3y ago
  > However, the Zimaboard specs claim to be using Realtek 8111H. Maybe they changed the design recently? D'oh! You are absolutely right, I am not sure from where I got Broadcom for ZimaBoard. Probably because I conflate BCM and RTL as both being "bleh" quality (OK for light in-band management, but no serious workloads)? Weak excuse for the inaccuracy, though.
  - Fischgericht 3y ago
    Well, if you pick a matching driver/kernel version, the 8111H work in a stable fashion. The hardware is mostly OK, the drivers are bad. You can even enable most offloading features without stuff exploding. On most distros, the 8111H works fine out of the box. I still miss the times of Intel 82574L & co, where you had BOTH working hardware AND working drivers.

Fischgericht 3y ago

Agreed, I'd never ever buy a product that has a Broadcom NIC or any other networking equipment based on Broadcom NICs. As an example, their 10G BCM53161 controllers have a power consumption of TWICE of the competing Realteks RTL8372. However, the Zimaboard specs claim to be using Realtek 8111H. Maybe they changed the design recently? Now, the 8111H also has a hell lot of issues, again recently with Linux Kernel 6.2, but it's nowhere as bad as the Broadcom stuff. For maybe $2 more they could have used some PROVEN (aka old) Intel NIC instead. (The recent Intel 2.5G I225-VI and 226-V ALSO have a bunch of problems/bugs, sigh) In general, quality of NIC chips has gone down during the last 10 years. If you want something to be used for critical/stable networking, you really have to do some research.

bcrl 3y ago
Intel NICs have had similar problems. One of my circa 2010 servers had the NIC lock up in such a way that passthru traffic to the IPMI was blocked preventing remote management access. I drove 3.5 hours to the data center to power cycle the system. Investigation eventually turned up an errata for exactly that issue. Suffice it to say that I decommissioned that server as soon as I could.
- T3OU-736 3y ago
  > Intel NICs have had similar problems. One of my circa 2010 servers had the NIC lock up in such a way that passthru traffic to the IPMI was blocked preventing remote management access. I drove 3.5 hours to the data center to power cycle the system. Investigation eventually turned up an errata for exactly that issue. Suffice it to say that I decommissioned that server as soon as I could. In the past life of being a SysAdmin, we strove for independent OOB NICs whenever possible specifically for this sort of edge case. And then used switched type PDUs (instead of just metered) so that remote power-cycling was possible for the eventuality that the IPMI controller had an OOM and no incantation of `ipmitool` would suffice (<caugh> iDRAC <caugh>). Sounds like you weren't able to get all those nice things. Unfortunate. Been there.
- Fischgericht 3y ago
  In-Band passthru is a nightmare. Even if your NIC is good, it can cause all sorts of problems. It's okayish if it's done via an Ethernet bridge, but that seems to be the exception. On the ASRock Rack boards I have recently sourced you suddenly get a new PCIe root bridge, and (10G) NIC throughput goes down the drain.
T3OU-736 3y ago
> However, the Zimaboard specs claim to be using Realtek 8111H. Maybe they changed the design recently? D'oh! You are absolutely right, I am not sure from where I got Broadcom for ZimaBoard. Probably because I conflate BCM and RTL as both being "bleh" quality (OK for light in-band management, but no serious workloads)? Weak excuse for the inaccuracy, though.
- Fischgericht 3y ago
  Well, if you pick a matching driver/kernel version, the 8111H work in a stable fashion. The hardware is mostly OK, the drivers are bad. You can even enable most offloading features without stuff exploding. On most distros, the 8111H works fine out of the box. I still miss the times of Intel 82574L & co, where you had BOTH working hardware AND working drivers.