7 ms·

OpenBSD Router Guide (2020)

160 points by rodrigo975 5y ago

francis-io 5y ago
I spent some time a while back using this guide, along with some ansible to setup a home router. I ended up putting it on the back burner for a few reasons. 1) DNS reflection was an issue for some internal services I host. 2) Connecting to my Virgin Media (uk) router and handling PPPoE was confusing and I struggled with the documentation. 3) Just in general, having an internet connection is really importaint to troubleshooting problems. Having my internet down while trying to fix things was just too inconvenient for me. In the end, I went back to PfSense. I have a bash script that backs up the config daily to s3. Was good enough to recover from a different issue recently. It's a real shame, I'd love to do this all my self but I may never pick this project back up.
- cpach 5y ago
  PfSense is not a bad fall-back IMHO (^_^)
  - AnthonyUK 5y ago
    Or OPNSense.
    - RostiB 5y ago
      moved from Pfsense to Opnsense, never looking back
      - isbjorn16 5y ago
        What were the compelling reasons for you to switch? Was it something Opnsense did better, or did Netgate just piss you off? I'm running pfsense now because I'm nothing if not lazy, but if opnsense makes things nicer in some way, I'd like to hear it!
- nicolaslem 5y ago
  I did exactly that last week with Debian 11. I got tired of IPFire (a linux distribution for routers, like pfSense). Since my only connectivity to the Internet is LTE, the most challenging part was bootstrapping the system without an Internet connection, after that it went very smoothly.
- Arch-TK 5y ago
  What Virgin Media modem+router are you using? I have a Superhub 3.0. In modem-only mode there's no PPPoE, it just rather transparently acts like you have an ethernet cable running to your ISP. Just putting dhcp in my hostname.em0 works exactly as intended and my router gets assigned the public IP address.
- SpaceInvader 5y ago
  I also had some issues with PPPoE, for some reason it did not worked for me, then suddenly it started to work with the very same config. That was very weird, but it still works fine. shark# cat /etc/hostname.pppoe0 inet 0.0.0.0 255.255.255.255 NONE \ pppoedev em0 authproto chap \ authname '<username>' authkey '<my_password>' up dest 0.0.0.1 !/sbin/route delete default !/sbin/route add default -ifp pppoe0 0.0.0.1 shark# cat /etc/hostname.em0 up shark#
Arch-TK 5y ago
Neat! I already have an OpenBSD firewall running on a PC Engines APU 2. Despite this, I still learned a few things from this guide (namely that urpf-failed already handles antispoof and that scrub shouldn't just be littered without any consideration). I was actually impressed at how well things perform despite the BKL of OpenBSD. I have only a 350Mbps down 35Mbps up connection and OpenBSD is nowhere near being a bottleneck.
- na85 5y ago
  Conversely, I tried running openbsd as a router on my ubiquiti edgerouter for my 950 megabit up/down connection and couldn't get even close to the performance of even the shitty ISP-provided router/ONT combo box.
  - elevation 5y ago
    As someone who had considered a similar setup, what model of edgerouter? Would you consider doing a writeup? What kind of speeds did you achieve?
    - na85 5y ago
      I get line speed with the Ubiquiti software installed. Here's a writeup I did last year but it doesn't touch on OpenBSD: https://0x85.org/ubiquiti.html https://0x85.org/ubiquiti.html I'm mostly over BSD in general. I don't know why, given that Debian exists and performs much better, I'd reach for BSD unless it had support for some esoteric hardware that Linux doesn't support.
hultner 5y ago
Neat guide but I'd much rather see a guide on how to set up a cost-effective 10Gbps WAN to LAN throughput router/gateway with NAT. Those of us building our own routers (I've done so since about 2006 when I got 1Gbps fibre installed for the first time) usually have a quite large cross-section with people on bleeding edge internet connections, and with (pro-)consumer 10Gbps internet and networking becoming more and more common a router capable of such is also more relevant. I see very little information about what's the minimal/cost effective hardware requirements capable of routing with NAT at wirespeed for these applications. Most of the content I read is either geared towards slower speeds (e.g. 1Gbps) or using overly beefy machines which are both power hungry and most likely overkill for the purpose. Something that would be interesting to see as well is high speed NAT routing performance comparison between Linux, NetBSD, OpenBSD and FreeBSD using Intel or other well supported 10Gbps nics.
- thedougd 5y ago
  I've found a few open source NAT implementations on DPDK, but several appears to be reference/demo implementations. This project looks interesting: https://vigor-nf.github.io/ https://vigor-nf.github.io/ DPDK network HBAs have been out for several years now and 25/50/100 is replacing 10 in the datacenter. Maybe there's some deals available on ebay? Quick glance, an Intel X520 is available for ~90 USD.
  - ay 5y ago
    VPP (http://fd.io http://fd.io) NAT plugins are used in various production deployments. Various NAT-related perf tests: https://docs.fd.io/csit/master/report/vpp_performance_tests/packet_throughput_graphs/nat44.html https://docs.fd.io/csit/master/report/vpp_performance_tests/... Disclaimer: I am involved with the project
- q3k 5y ago
  0. Accept that for NAT on home gateway you really don't need wirespeed at minimal MTU, as the majority of your data transfers will be over TCP at max MSS/MTU. 1. Get a Xeon D box (like a Supermicro X10SDV-TLN4F, TDP 45W). 2. Install any Linux distribution on it and just use iptables/nftables. It's good enough. 3. You now have a normal Linux box with normal userspace networking, feel free to also run some more services on it without having to deal with DPDK getting in your way. DPDK is nice and all if you absolutely need to scream through tiny packets at multi-dozen-gigabit speeds, but you almost certainly don't need it at home for Netflix and seeding Ubuntu ISOs.
bluedino 5y ago
Are there any open-source, "next-generation" firewalls, running on OpenBSD? SD-WAN, DLP, Application-aware filtering, etc
- FuriouslyAdrift 5y ago
  I don't know of any big ones that are still going. OPNSense runs on HardenedBSD (a FreeBSD fork) and is excellent. FreeBSD and/or NetBSD is/was used by Juniper, Sandvine, Dell (Force10). I think the main blocker for OpenBSD in the commercial market is the inability to load binary blob drivers without maintaining a custom kernel.
  - LargoLasskhyfv 5y ago
    Next release of OPNSense will be rebased on 'stock' FreeBSD. The current release is the last to be based on HardenedBSD. This is known since April https://forum.opnsense.org/index.php?topic=22761.0 https://forum.opnsense.org/index.php?topic=22761.0 and confirmed there https://forum.opnsense.org/index.php?topic=24112.0 https://forum.opnsense.org/index.php?topic=24112.0
- 1MachineElf 5y ago
  This is an interesting question. One would think, since OpenBSD is BSD licensed, it would be more likely to have vendorized forks available on the marketplace. Companies being obliged to share their code is allegedly a disadvantage of GPL-licenced Linux. But with OpenBSD, there are no prominent commercial forks. The one "active" OpenBSD fork today is HyperbolaBSD, and it hasn't made a single release yet after 2 years. Previous attempts at OpenBSD forks have failed (Bitrig, ÆrieBSD). Somehow, OpenBSD appears to be less fork-able than other BSDs. Why does the OpenBSD project subvert our expectations of a BSD-licenced OS? I'm asking as someone who had a positive experience setting up an OpenBSD router and VPN, but not much beyond that.
  - LargoLasskhyfv 5y ago
    Maybe because the founder and team is known for zero tolerance toward BS, gives a shit about percieved 'needs' of shitty 'bizniz', and drama.
candl 5y ago
Is OpenBSD still largely single-threaded or have there been SMP improvements in the network stack over the years? The feature set OpenBSD has is impressive, but is there a large gap in networking perf compared to Linux/FreeBSD?
- eu 5y ago
  From my $work computer: $ uname -a OpenBSD XXXXX 6.9 GENERIC.MP#3 amd64 $top load averages: 0.83, 0.62, 0.44 94 processes: 93 idle, 1 on processor CPU00 states: 0.5% user, 0.0% nice, 0.2% sys, 0.3% spin, 0.6% intr, 98.4% idle CPU02 states: 6.5% user, 0.0% nice, 1.9% sys, 0.4% spin, 0.0% intr, 91.1% idle CPU04 states: 4.9% user, 0.0% nice, 1.6% sys, 0.3% spin, 0.0% intr, 93.1% idle CPU06 states: 4.0% user, 0.0% nice, 1.1% sys, 0.3% spin, 0.0% intr, 94.6% idle CPU08 states: 1.4% user, 0.0% nice, 0.6% sys, 0.2% spin, 0.0% intr, 97.8% idle CPU10 states: 1.4% user, 0.0% nice, 0.5% sys, 0.2% spin, 0.0% intr, 98.0% idle CPU12 states: 0.6% user, 0.0% nice, 0.3% sys, 0.1% spin, 0.0% intr, 99.0% idle CPU14 states: 0.7% user, 0.0% nice, 0.3% sys, 0.1% spin, 0.0% intr, 98.8% idle Memory: Real: 3292M/5988M act/tot Free: 9109M Cache: 1692M Swap: 0K/15G
  - tadbit 5y ago
    The person you're responding to was asking about network performance. I imagine specifically regarding pf (packet filter, the firewall component). To the grandparent: PF is still single threaded. If you had performance issues with that before you may still have them, but CPU improvements over time may have negated that impact. It's worth trying it out again.
madjam002 5y ago
My favourite is currently NixOS for a router, entire configuration (system packages, version, firewall rules, interfaces, VLANs, etc) is declarative and can be deployed with automatic rollbacks in the event of misconfiguration.
- inapis 5y ago
  That sounds very interesting. Have you documented it somewhere?
  - ochoseis 5y ago
    Here’s a really simple config I started with (not mine, just inspiration): https://github.com/jgillich/nixos/blob/master/roles/router.nix https://github.com/jgillich/nixos/blob/master/roles/router.n...
- ochoseis 5y ago
  This is what I’m working on right now too, built around an APU board. I’m also wrestling with hostapd to get WiFi working as well, just because. Once it’s working I’ll try and write it up.
- 0xCMP 5y ago
  I also love this. I ran into issues trying to get it working with extra very strict kernel security features enabled, but even then I eventually figured it out. Was really nice to use.
teleforce 5y ago
Anyone know similar up-to-date comprehensive guide for Linux? This book by Tony Mancill used to be an excellent guide for Linux routers but now after 20 years it is already obselete [1]. [1]https://www.amazon.com/Linux-Routers-Primer-Network-Administrators/dp/0130090263 https://www.amazon.com/Linux-Routers-Primer-Network-Administ...
Datenstrom 5y ago
OpenBSD on the APU2 has been the most reliable, solid, and lowest maintenance router I have ever owned. The thing never needs to be restarted, never hiccups, and just is really rock solid. Highly recommend, just wish there was a good wireless AP solution too.
120photo 5y ago
Building a OpenBSD router has always been one of those projects in the back of my head but just never had the time. I plan on upgrading my home network in the next year and am considering MikroTik / RouterOS or a Ubiquity router. Anyone have experience with any of those and a custom OpenBSD build? What would be the advantage of going OpenBSD and is it worth it for your setup?

160 points by rodrigo975 5y ago

I spent some time a while back using this guide, along with some ansible to setup a home router. I ended up putting it on the back burner for a few reasons. 1) DNS reflection was an issue for some internal services I host. 2) Connecting to my Virgin Media (uk) router and handling PPPoE was confusing and I struggled with the documentation. 3) Just in general, having an internet connection is really importaint to troubleshooting problems. Having my internet down while trying to fix things was just too inconvenient for me. In the end, I went back to PfSense. I have a bash script that backs up the config daily to s3. Was good enough to recover from a different issue recently. It's a real shame, I'd love to do this all my self but I may never pick this project back up.

cpach 5y ago
PfSense is not a bad fall-back IMHO (^_^)
- AnthonyUK 5y ago
  Or OPNSense.
  - RostiB 5y ago
    moved from Pfsense to Opnsense, never looking back
    - isbjorn16 5y ago
      What were the compelling reasons for you to switch? Was it something Opnsense did better, or did Netgate just piss you off? I'm running pfsense now because I'm nothing if not lazy, but if opnsense makes things nicer in some way, I'd like to hear it!
nicolaslem 5y ago
I did exactly that last week with Debian 11. I got tired of IPFire (a linux distribution for routers, like pfSense). Since my only connectivity to the Internet is LTE, the most challenging part was bootstrapping the system without an Internet connection, after that it went very smoothly.
Arch-TK 5y ago
What Virgin Media modem+router are you using? I have a Superhub 3.0. In modem-only mode there's no PPPoE, it just rather transparently acts like you have an ethernet cable running to your ISP. Just putting dhcp in my hostname.em0 works exactly as intended and my router gets assigned the public IP address.
SpaceInvader 5y ago
I also had some issues with PPPoE, for some reason it did not worked for me, then suddenly it started to work with the very same config. That was very weird, but it still works fine. shark# cat /etc/hostname.pppoe0 inet 0.0.0.0 255.255.255.255 NONE \ pppoedev em0 authproto chap \ authname '<username>' authkey '<my_password>' up dest 0.0.0.1 !/sbin/route delete default !/sbin/route add default -ifp pppoe0 0.0.0.1 shark# cat /etc/hostname.em0 up shark#

Arch-TK 5y ago

Neat! I already have an OpenBSD firewall running on a PC Engines APU 2. Despite this, I still learned a few things from this guide (namely that urpf-failed already handles antispoof and that scrub shouldn't just be littered without any consideration). I was actually impressed at how well things perform despite the BKL of OpenBSD. I have only a 350Mbps down 35Mbps up connection and OpenBSD is nowhere near being a bottleneck.

na85 5y ago
Conversely, I tried running openbsd as a router on my ubiquiti edgerouter for my 950 megabit up/down connection and couldn't get even close to the performance of even the shitty ISP-provided router/ONT combo box.
- elevation 5y ago
  As someone who had considered a similar setup, what model of edgerouter? Would you consider doing a writeup? What kind of speeds did you achieve?
  - na85 5y ago
    I get line speed with the Ubiquiti software installed. Here's a writeup I did last year but it doesn't touch on OpenBSD: https://0x85.org/ubiquiti.html https://0x85.org/ubiquiti.html I'm mostly over BSD in general. I don't know why, given that Debian exists and performs much better, I'd reach for BSD unless it had support for some esoteric hardware that Linux doesn't support.

hultner 5y ago

Neat guide but I'd much rather see a guide on how to set up a cost-effective 10Gbps WAN to LAN throughput router/gateway with NAT. Those of us building our own routers (I've done so since about 2006 when I got 1Gbps fibre installed for the first time) usually have a quite large cross-section with people on bleeding edge internet connections, and with (pro-)consumer 10Gbps internet and networking becoming more and more common a router capable of such is also more relevant. I see very little information about what's the minimal/cost effective hardware requirements capable of routing with NAT at wirespeed for these applications. Most of the content I read is either geared towards slower speeds (e.g. 1Gbps) or using overly beefy machines which are both power hungry and most likely overkill for the purpose. Something that would be interesting to see as well is high speed NAT routing performance comparison between Linux, NetBSD, OpenBSD and FreeBSD using Intel or other well supported 10Gbps nics.

thedougd 5y ago
I've found a few open source NAT implementations on DPDK, but several appears to be reference/demo implementations. This project looks interesting: https://vigor-nf.github.io/ https://vigor-nf.github.io/ DPDK network HBAs have been out for several years now and 25/50/100 is replacing 10 in the datacenter. Maybe there's some deals available on ebay? Quick glance, an Intel X520 is available for ~90 USD.
- ay 5y ago
  VPP (http://fd.io http://fd.io) NAT plugins are used in various production deployments. Various NAT-related perf tests: https://docs.fd.io/csit/master/report/vpp_performance_tests/packet_throughput_graphs/nat44.html https://docs.fd.io/csit/master/report/vpp_performance_tests/... Disclaimer: I am involved with the project
q3k 5y ago
0. Accept that for NAT on home gateway you really don't need wirespeed at minimal MTU, as the majority of your data transfers will be over TCP at max MSS/MTU. 1. Get a Xeon D box (like a Supermicro X10SDV-TLN4F, TDP 45W). 2. Install any Linux distribution on it and just use iptables/nftables. It's good enough. 3. You now have a normal Linux box with normal userspace networking, feel free to also run some more services on it without having to deal with DPDK getting in your way. DPDK is nice and all if you absolutely need to scream through tiny packets at multi-dozen-gigabit speeds, but you almost certainly don't need it at home for Netflix and seeding Ubuntu ISOs.

bluedino 5y ago

Are there any open-source, "next-generation" firewalls, running on OpenBSD? SD-WAN, DLP, Application-aware filtering, etc

FuriouslyAdrift 5y ago
I don't know of any big ones that are still going. OPNSense runs on HardenedBSD (a FreeBSD fork) and is excellent. FreeBSD and/or NetBSD is/was used by Juniper, Sandvine, Dell (Force10). I think the main blocker for OpenBSD in the commercial market is the inability to load binary blob drivers without maintaining a custom kernel.
- LargoLasskhyfv 5y ago
  Next release of OPNSense will be rebased on 'stock' FreeBSD. The current release is the last to be based on HardenedBSD. This is known since April https://forum.opnsense.org/index.php?topic=22761.0 https://forum.opnsense.org/index.php?topic=22761.0 and confirmed there https://forum.opnsense.org/index.php?topic=24112.0 https://forum.opnsense.org/index.php?topic=24112.0
1MachineElf 5y ago
This is an interesting question. One would think, since OpenBSD is BSD licensed, it would be more likely to have vendorized forks available on the marketplace. Companies being obliged to share their code is allegedly a disadvantage of GPL-licenced Linux. But with OpenBSD, there are no prominent commercial forks. The one "active" OpenBSD fork today is HyperbolaBSD, and it hasn't made a single release yet after 2 years. Previous attempts at OpenBSD forks have failed (Bitrig, ÆrieBSD). Somehow, OpenBSD appears to be less fork-able than other BSDs. Why does the OpenBSD project subvert our expectations of a BSD-licenced OS? I'm asking as someone who had a positive experience setting up an OpenBSD router and VPN, but not much beyond that.
- LargoLasskhyfv 5y ago
  Maybe because the founder and team is known for zero tolerance toward BS, gives a shit about percieved 'needs' of shitty 'bizniz', and drama.

candl 5y ago

Is OpenBSD still largely single-threaded or have there been SMP improvements in the network stack over the years? The feature set OpenBSD has is impressive, but is there a large gap in networking perf compared to Linux/FreeBSD?

eu 5y ago
From my $work computer: $ uname -a OpenBSD XXXXX 6.9 GENERIC.MP#3 amd64 $top load averages: 0.83, 0.62, 0.44 94 processes: 93 idle, 1 on processor CPU00 states: 0.5% user, 0.0% nice, 0.2% sys, 0.3% spin, 0.6% intr, 98.4% idle CPU02 states: 6.5% user, 0.0% nice, 1.9% sys, 0.4% spin, 0.0% intr, 91.1% idle CPU04 states: 4.9% user, 0.0% nice, 1.6% sys, 0.3% spin, 0.0% intr, 93.1% idle CPU06 states: 4.0% user, 0.0% nice, 1.1% sys, 0.3% spin, 0.0% intr, 94.6% idle CPU08 states: 1.4% user, 0.0% nice, 0.6% sys, 0.2% spin, 0.0% intr, 97.8% idle CPU10 states: 1.4% user, 0.0% nice, 0.5% sys, 0.2% spin, 0.0% intr, 98.0% idle CPU12 states: 0.6% user, 0.0% nice, 0.3% sys, 0.1% spin, 0.0% intr, 99.0% idle CPU14 states: 0.7% user, 0.0% nice, 0.3% sys, 0.1% spin, 0.0% intr, 98.8% idle Memory: Real: 3292M/5988M act/tot Free: 9109M Cache: 1692M Swap: 0K/15G
- tadbit 5y ago
  The person you're responding to was asking about network performance. I imagine specifically regarding pf (packet filter, the firewall component). To the grandparent: PF is still single threaded. If you had performance issues with that before you may still have them, but CPU improvements over time may have negated that impact. It's worth trying it out again.

madjam002 5y ago

My favourite is currently NixOS for a router, entire configuration (system packages, version, firewall rules, interfaces, VLANs, etc) is declarative and can be deployed with automatic rollbacks in the event of misconfiguration.

inapis 5y ago
That sounds very interesting. Have you documented it somewhere?
- ochoseis 5y ago
  Here’s a really simple config I started with (not mine, just inspiration): https://github.com/jgillich/nixos/blob/master/roles/router.nix https://github.com/jgillich/nixos/blob/master/roles/router.n...
ochoseis 5y ago
This is what I’m working on right now too, built around an APU board. I’m also wrestling with hostapd to get WiFi working as well, just because. Once it’s working I’ll try and write it up.
0xCMP 5y ago
I also love this. I ran into issues trying to get it working with extra very strict kernel security features enabled, but even then I eventually figured it out. Was really nice to use.

teleforce 5y ago

Anyone know similar up-to-date comprehensive guide for Linux? This book by Tony Mancill used to be an excellent guide for Linux routers but now after 20 years it is already obselete [1]. [1]https://www.amazon.com/Linux-Routers-Primer-Network-Administrators/dp/0130090263 https://www.amazon.com/Linux-Routers-Primer-Network-Administ...

Datenstrom 5y ago

OpenBSD on the APU2 has been the most reliable, solid, and lowest maintenance router I have ever owned. The thing never needs to be restarted, never hiccups, and just is really rock solid. Highly recommend, just wish there was a good wireless AP solution too.

120photo 5y ago

Building a OpenBSD router has always been one of those projects in the back of my head but just never had the time. I plan on upgrading my home network in the next year and am considering MikroTik / RouterOS or a Ubiquity router. Anyone have experience with any of those and a custom OpenBSD build? What would be the advantage of going OpenBSD and is it worth it for your setup?