6 ms·

I’ve been thinking about doing a Raspberry Pi CM4 with OpenWrt. Jeff Geerling recently did a post about two of the IO boards for it: https://www.jeffgeerling.co

I’ve been thinking about doing a Raspberry Pi CM4 with OpenWrt. Jeff Geerling recently did a post about two of the IO boards for it: https://www.jeffgeerling.com/blog/2021/two-tiny-dual-gigabit-raspberry-pi-cm4-routers https://www.jeffgeerling.com/blog/2021/two-tiny-dual-gigabit... Would anybody recommend that sort of setup over a Mikrotik Hex or Ubiquiti Edgerouter X?

stock_toaster 5y ago
I moved from an edgerouter4[1] to openwrt on a protectli fw4b. On my home gigabit fiber (centurylink) link, it has been great so far (performance, throughput, etc)! [1]: Ubiquiti has had packet processing issues in the 2.x firmware series for what seems like forever, and lack of updates lately has prompted me to move to something that is arguably better anyway.
stragies 5y ago
Both built-in interfaces of the RPI4 don't support MTU over 1500. That rules out many advanced usage scenarios, I think.
- geerlingguy 5y ago
  They can be modified, though not necessarily in a typical way. Also, CM4 implementations vary, and most interfaces on 3rd party boards can have MTU set as normal.
  - stragies 5y ago
    Hi, thanks for that correction. Do you have a keyword or 2 to google for "They can be modified, though not necessarily in a typical way." Or is that only or the CM, and not the normal RPI4? I only found some quick&dirty patch, that seems to make MTU up to 2000 functional. But since the patch aims for 9018, and only gets 2000, that does not seem like the "right" solution.
    - geerlingguy 5y ago
      You can do it if you compile in a kernel patch on the regular Pi 4, I believe: https://www.jeffgeerling.com/blog/2020/setting-9000-mtu-jumbo-frames-on-raspberry-pi-os https://www.jeffgeerling.com/blog/2020/setting-9000-mtu-jumb...
ComputerGuru 5y ago
I can’t recommend the CM4 over anything with an actual switch built into it with network fabric providing CPU-free packet switching and forwarding. You also honestly don’t want something as important as your actual internet gateway to be some hacky thing that might or might not be supported in a few months or years, unless it's just for experimentation in which case knock yourself out.
- cyounkins 5y ago
  Unfortunately I don't think there are any implementations of CPU-free QoS, which you really want at the network boundary.
  - LargoLasskhyfv 5y ago
    That really depends on the used SOC, or how well its hardware features are supported under Linux, by non-BLOBs, preferrably. Anything MARVELL should mostly do, usually. But that's not the cheapest segment anymore, so the people are hurting themselves by buying the common plastic trash. Or use DD-WRT instead of OpenWRT, because they give a shit about binary-blobs and integrate that however they see fit, because they prefer to be able to use everything the hardware supports, instead of having an almost brick running open-source.
    - cyounkins 5y ago
      I don't think any devices supported by OpenWRT, DD-WRT, VyOS, pfSense, etc support hardware-accelerated routing with QoS. If you know of one, let me know! It is not always clear, but enabling QoS usually disables hardware offloading, for example with Ubiquiti devices[1] "Traffic to which QoS policies have been applied cannot also take advantage of the EdgeRouter's Offloading feature". The same is true for open source even with binary blobs. The chips simply can't do it. Using QoS and implicitly disabling hardware offloading significantly drops the throughput as it becomes CPU bound. "Without offloading enabled, IPv4 traffic will be routed via the CPU and will be limited to around 300Mbps on the EdgeRouter Lite (ERLite-3). With offloading enabled, the throughput will be about 950Mbps." [1] [1] https://help.ui.com/hc/en-us/articles/115006567467 https://help.ui.com/hc/en-us/articles/115006567467
jvolkman 5y ago
I found a passively cooled x86 box that I run openwrt on. It's got 6 Intel gigabit NICs and I use Proxmox to virtualize guests. OpenWrt is one of those guests, as is Home Assistant which controls various home automation things. I posted about it on Reddit a while ago: https://www.reddit.com/r/homelab/comments/hzvfih/new_router_i58365u_quad_core_6_intel_nics https://www.reddit.com/r/homelab/comments/hzvfih/new_router_... At the time it had much better hardware at a similar price to more popular brands like Protectli and Qotom. Not sure what the scene is currently, but there a bunch of similar offerings on AliExpress.
- zbrozek 5y ago
  Why did you choose x86 openwrt over opnsense? I chose opnsense because it never occurred to me to run openwrt on not-a-consumer-router and now I'm wondering if I screwed up. Like you, I run a single x86 box with a handful of guests for random stuff - including Home Assistant!
  - jvolkman 5y ago
    I did try opnsense, but found that I wanted wireguard more than any of the features it offered. At the time wireguard wasn't a built-in option, so went back to OpenWrt which I was already familiar with anyway from running it on Linksys routers.
    - zbrozek 5y ago
      Makes sense! I'm using wireguard on opnsense (but it's a newish feature) and so far it's been a great experience.
    - circularfoyers 5y ago
      I had a similar experience and ended up doing the same. OPNsense does include a userspace implementation of Wireguard, but is obviously inferior to what Linux already has. It ended up being that, how poorly the initial kernel space implementation was handled for FreeBSD and that I wanted Cake for QoS, which made me move back to OpenWRT. I do want to ask though, has there been any downsides to using OpenWRT on x86?
zamadatix 5y ago
If you're looking for a router that does NAT you're not going to beat a purpose built hardware board in terms of perf per dollar, especially only latency and jitter of the NAT. If you're looking for a network connected server that happens to do a lot of packet processing you can go the rpi route but I'd still front the internet and rest of the home internet off a small cheap hardware box like the Hex.
antattack 5y ago
If you want something fast and inexpensive than wait for OpenWRT build for Redmi AX6 / AX3600 WIFI routers. For now only manual build is available and some have to reboot from time to time due to memory leak. https://forum.openwrt.org/t/adding-openwrt-support-for-xiaomi-ax3600/55049/4058 https://forum.openwrt.org/t/adding-openwrt-support-for-xiaom...
scottlamb 5y ago
One downside to the Raspberry Pi (CM)4 is that it doesn't have hardware crypto. So it'd make a poor VPN machine. But otherwise I think it'd be fine. Some people are happy just using a regular Pi 4 connected to a VLAN-capable switch or with a USB adapter for the second NIC. I recently bought a GL.iNet Brume. It's a nice tiny ARM machine I use as a wired-only router. /proc/cpuinfo confirms it supports hardware crypto... Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid ...although I haven't set up a VPN on it yet. Downsides: the WAN and LAN0/LAN1 ports are all the same switch with only one network interface, so it'd be impossible to achieve >500Mbps full-duplex. (It's never come up for me; my Internet connection's too slow.) It's also a little pricy for what it is. I might have tried the FriendlyELEC NanoPI R4S instead, which actually has two separate NICs. But if I'd ordered that I'd still be waiting for it... One thing I don't care about: hardware NAT/routing offload. I think that's often unsupported by OpenWRT, and it's incompatible with SQM.
- squarefoot 5y ago
  Not sure if it supports all the features you need, anyway you might find interesting the PCEngines APU2 boards. https://pcengines.ch/apu2.htm https://pcengines.ch/apu2.htm
  - scottlamb 5y ago
    Those are neat looking machines. I saw them and iirc quickly ruled them out because they're not available now.
  - cycomanic 5y ago
    Anyone has experiences with the complete build from https://teklager.se/en/products/routers/ https://teklager.se/en/products/routers/ They look very nice (although unavailable till January)
- stragies 5y ago
  Concerning HW NAT/Offload support: the DSA support is (iiuc) supposed to be the framework, through which the offload functionalities will be implemented. There is a pull-request for DSA even for architectures as far back as ath79. I haven't read into the specifics yet, but it sounds like nft-tables configuration will be translated to a eBPF program, that will then hook into network flows, and tell the switching HW/NPE over some side channel which accelerations to use and shortcuts to take. But do I really want to trust a "Black Box Network Processing Engine" bolted onto my CPU, that can route packets entirely outside the realm of control of the main CPU? I always wonder about intentional or unintentional "in-band debug features", when people mention, that they run their home-lan on the same switch as the WAN-Uplink. I mean, is there really a HW vendor, that we trust to get the switching logic 100% correct/secure/performant?
Wubdidu 5y ago
Probably depends on how complex your setup will be. We are running a normal 1GB Pi4 with OpenWRT (trunk until now) for quite a while on a 1gbit down/500mbit up fibre uplink with SQM against bufferbloat. Unifi switches handle all VLAN related work, fibre uplink is on VLAN 7 mandated by the carrier, so PPPoE works with only one LAN port on the Pi4. Wifi is also unifi hardware. Thanks to full duplex, clients are able to use the fibre uplink without slowdown. VPN is managed by a separate system, though we had some successful experiments with WireGuard. Both a colleague and I are running the exact same setup at home (with fibre uplink and VDSL). It's a very capable and power efficient setup. I'd definitely go with a CM4+dual NIC board today.

stock_toaster 5y ago

I moved from an edgerouter4[1] to openwrt on a protectli fw4b. On my home gigabit fiber (centurylink) link, it has been great so far (performance, throughput, etc)! [1]: Ubiquiti has had packet processing issues in the 2.x firmware series for what seems like forever, and lack of updates lately has prompted me to move to something that is arguably better anyway.

stragies 5y ago

Both built-in interfaces of the RPI4 don't support MTU over 1500. That rules out many advanced usage scenarios, I think.

geerlingguy 5y ago
They can be modified, though not necessarily in a typical way. Also, CM4 implementations vary, and most interfaces on 3rd party boards can have MTU set as normal.
- stragies 5y ago
  Hi, thanks for that correction. Do you have a keyword or 2 to google for "They can be modified, though not necessarily in a typical way." Or is that only or the CM, and not the normal RPI4? I only found some quick&dirty patch, that seems to make MTU up to 2000 functional. But since the patch aims for 9018, and only gets 2000, that does not seem like the "right" solution.
  - geerlingguy 5y ago
    You can do it if you compile in a kernel patch on the regular Pi 4, I believe: https://www.jeffgeerling.com/blog/2020/setting-9000-mtu-jumbo-frames-on-raspberry-pi-os https://www.jeffgeerling.com/blog/2020/setting-9000-mtu-jumb...

ComputerGuru 5y ago

I can’t recommend the CM4 over anything with an actual switch built into it with network fabric providing CPU-free packet switching and forwarding. You also honestly don’t want something as important as your actual internet gateway to be some hacky thing that might or might not be supported in a few months or years, unless it's just for experimentation in which case knock yourself out.

cyounkins 5y ago
Unfortunately I don't think there are any implementations of CPU-free QoS, which you really want at the network boundary.
- LargoLasskhyfv 5y ago
  That really depends on the used SOC, or how well its hardware features are supported under Linux, by non-BLOBs, preferrably. Anything MARVELL should mostly do, usually. But that's not the cheapest segment anymore, so the people are hurting themselves by buying the common plastic trash. Or use DD-WRT instead of OpenWRT, because they give a shit about binary-blobs and integrate that however they see fit, because they prefer to be able to use everything the hardware supports, instead of having an almost brick running open-source.
  - cyounkins 5y ago
    I don't think any devices supported by OpenWRT, DD-WRT, VyOS, pfSense, etc support hardware-accelerated routing with QoS. If you know of one, let me know! It is not always clear, but enabling QoS usually disables hardware offloading, for example with Ubiquiti devices[1] "Traffic to which QoS policies have been applied cannot also take advantage of the EdgeRouter's Offloading feature". The same is true for open source even with binary blobs. The chips simply can't do it. Using QoS and implicitly disabling hardware offloading significantly drops the throughput as it becomes CPU bound. "Without offloading enabled, IPv4 traffic will be routed via the CPU and will be limited to around 300Mbps on the EdgeRouter Lite (ERLite-3). With offloading enabled, the throughput will be about 950Mbps." [1] [1] https://help.ui.com/hc/en-us/articles/115006567467 https://help.ui.com/hc/en-us/articles/115006567467

jvolkman 5y ago

I found a passively cooled x86 box that I run openwrt on. It's got 6 Intel gigabit NICs and I use Proxmox to virtualize guests. OpenWrt is one of those guests, as is Home Assistant which controls various home automation things. I posted about it on Reddit a while ago: https://www.reddit.com/r/homelab/comments/hzvfih/new_router_i58365u_quad_core_6_intel_nics https://www.reddit.com/r/homelab/comments/hzvfih/new_router_... At the time it had much better hardware at a similar price to more popular brands like Protectli and Qotom. Not sure what the scene is currently, but there a bunch of similar offerings on AliExpress.

zbrozek 5y ago
Why did you choose x86 openwrt over opnsense? I chose opnsense because it never occurred to me to run openwrt on not-a-consumer-router and now I'm wondering if I screwed up. Like you, I run a single x86 box with a handful of guests for random stuff - including Home Assistant!
- jvolkman 5y ago
  I did try opnsense, but found that I wanted wireguard more than any of the features it offered. At the time wireguard wasn't a built-in option, so went back to OpenWrt which I was already familiar with anyway from running it on Linksys routers.
  - zbrozek 5y ago
    Makes sense! I'm using wireguard on opnsense (but it's a newish feature) and so far it's been a great experience.
  - circularfoyers 5y ago
    I had a similar experience and ended up doing the same. OPNsense does include a userspace implementation of Wireguard, but is obviously inferior to what Linux already has. It ended up being that, how poorly the initial kernel space implementation was handled for FreeBSD and that I wanted Cake for QoS, which made me move back to OpenWRT. I do want to ask though, has there been any downsides to using OpenWRT on x86?

zamadatix 5y ago

If you're looking for a router that does NAT you're not going to beat a purpose built hardware board in terms of perf per dollar, especially only latency and jitter of the NAT. If you're looking for a network connected server that happens to do a lot of packet processing you can go the rpi route but I'd still front the internet and rest of the home internet off a small cheap hardware box like the Hex.

antattack 5y ago

If you want something fast and inexpensive than wait for OpenWRT build for Redmi AX6 / AX3600 WIFI routers. For now only manual build is available and some have to reboot from time to time due to memory leak. https://forum.openwrt.org/t/adding-openwrt-support-for-xiaomi-ax3600/55049/4058 https://forum.openwrt.org/t/adding-openwrt-support-for-xiaom...

scottlamb 5y ago

One downside to the Raspberry Pi (CM)4 is that it doesn't have hardware crypto. So it'd make a poor VPN machine. But otherwise I think it'd be fine. Some people are happy just using a regular Pi 4 connected to a VLAN-capable switch or with a USB adapter for the second NIC. I recently bought a GL.iNet Brume. It's a nice tiny ARM machine I use as a wired-only router. /proc/cpuinfo confirms it supports hardware crypto... Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid ...although I haven't set up a VPN on it yet. Downsides: the WAN and LAN0/LAN1 ports are all the same switch with only one network interface, so it'd be impossible to achieve >500Mbps full-duplex. (It's never come up for me; my Internet connection's too slow.) It's also a little pricy for what it is. I might have tried the FriendlyELEC NanoPI R4S instead, which actually has two separate NICs. But if I'd ordered that I'd still be waiting for it... One thing I don't care about: hardware NAT/routing offload. I think that's often unsupported by OpenWRT, and it's incompatible with SQM.

squarefoot 5y ago
Not sure if it supports all the features you need, anyway you might find interesting the PCEngines APU2 boards. https://pcengines.ch/apu2.htm https://pcengines.ch/apu2.htm
- scottlamb 5y ago
  Those are neat looking machines. I saw them and iirc quickly ruled them out because they're not available now.
- cycomanic 5y ago
  Anyone has experiences with the complete build from https://teklager.se/en/products/routers/ https://teklager.se/en/products/routers/ They look very nice (although unavailable till January)
stragies 5y ago
Concerning HW NAT/Offload support: the DSA support is (iiuc) supposed to be the framework, through which the offload functionalities will be implemented. There is a pull-request for DSA even for architectures as far back as ath79. I haven't read into the specifics yet, but it sounds like nft-tables configuration will be translated to a eBPF program, that will then hook into network flows, and tell the switching HW/NPE over some side channel which accelerations to use and shortcuts to take. But do I really want to trust a "Black Box Network Processing Engine" bolted onto my CPU, that can route packets entirely outside the realm of control of the main CPU? I always wonder about intentional or unintentional "in-band debug features", when people mention, that they run their home-lan on the same switch as the WAN-Uplink. I mean, is there really a HW vendor, that we trust to get the switching logic 100% correct/secure/performant?

Wubdidu 5y ago

Probably depends on how complex your setup will be. We are running a normal 1GB Pi4 with OpenWRT (trunk until now) for quite a while on a 1gbit down/500mbit up fibre uplink with SQM against bufferbloat. Unifi switches handle all VLAN related work, fibre uplink is on VLAN 7 mandated by the carrier, so PPPoE works with only one LAN port on the Pi4. Wifi is also unifi hardware. Thanks to full duplex, clients are able to use the fibre uplink without slowdown. VPN is managed by a separate system, though we had some successful experiments with WireGuard. Both a colleague and I are running the exact same setup at home (with fibre uplink and VDSL). It's a very capable and power efficient setup. I'd definitely go with a CM4+dual NIC board today.