11 ms·
Moving from OpenBSD to FreeBSD for firewalls
- 0xWTF 11mo agoI don't understand why this has 29 points and no comments. What's so amazing about this?
- wslh 11mo agoI imagine a near future where TCP/IP stacks, and device drivers are interchangeable between operating systems. In Linux, NDISWrapper [1] enables to use Windows drivers in Linux but it's a wrapper (with all due respect to this project). [1] https://en.wikipedia.org/wiki/NDISwrapper https://en.wikipedia.org/wiki/NDISwrapper
- awesome_dude 11mo agoMicrosoft started out with BSD's TCP/IP stack, but dropped it for their own (back in Windows 3.5 apparently - https://news.ycombinator.com/item?id=41495551 https://news.ycombinator.com/item?id=41495551)
- EvanAnderson 11mo agoAdam Barr, formerly with Microsoft, goes into some detail about it here: https://web.archive.org/web/20051114154320/http://www.kuro5hin.org/?op=displaystory%3Bsid%3D2001%2F6%2F19%2F05641%2F7357 https://web.archive.org/web/20051114154320/http://www.kuro5h...
- zokier 11mo agoYou mean like DPDK?
- shanemhansen 11mo agoI'd think something like Rump Kernel's is a closer analogue: https://en.wikipedia.org/wiki/Rump_kernel https://en.wikipedia.org/wiki/Rump_kernel
- spragl 11mo agoThat sent me looking it up. It seems that NetBSD, as the only one, has a rump kernel, but it also looks like work on it stagnated around 10 years ago. That could be because the guy doing a thesis on them, moved on. There is quite some bitrot when following links. Do you know what happened? Were they a failure? Maybe they were surpassed by other OS architectures?
- miladyincontrol 11mo agoSorta, but only with ancient windows XP drivers. It was a useful stopgap of it's era but linux networking drivers have more than caught up in the meantime.
- jmclnx 11mo agoFor me, the only drawback for corporations is the 6 month upgrade. There is no LTS on OpenBSD. I use OpenBSD as a workstation and it works great, but in a production environment I doubt I would use OpenBSD for critical items, mainly because no LTS. It is a sad state of affairs because Companies do not want nor will want a system you need to upgrade so often even if its security very good.
- rootnod3 11mo agoOn the other hand though, updates on OpenBSD are the most painless updates I have ever done. I am more concerned about it's usage of UFS instead of something more robust for drives.
- kstrauser 11mo agoI'm grossly generalizing here, but it seems like OpenBSD boxes seem to be commonly used for the sorts of things that don't write a lot of data to local drives, except maybe logfiles. You can obviously use it for fileservers and such but I don't recall ever seeing that in the wild. So in that situation, UFS is fine. (IMO it's fine for heavier-write cases, too. It's just especially alright for the common deployment case where it's practically read-only anyway.)
- SoftTalker 11mo agoI've used it as a mail server, a web server, and a database (postgres) server. It's also my main desktop OS. Did/does fine, but I never really stressed it. I would certainly welcome a more capable filesystem option, as well as something like logical volumes, but I can't say that ufs has ever failed me. You'll definitely want to have it on a UPS to avoid some potentially long and sometimes manual intervention on fscks after a power failure. And of course, backups for anything important.
- traceroute66 11mo ago> updates on OpenBSD are the most painless updates I have ever done I see we have a post-syspatch (6.1 - 2017), post-sysupgrade (6.6 - 2019) OpenBSD user in our midsts. ;D You are positively a newbie in the OpenBSD world ! Some of us are old enough to remember when OpenBSD updates were a complete pain in the ass in involving downloading shit to /usr/src and compiling it yourself !
- awesome_dude 11mo ago> There are some things about FreeBSD that we're not entirely enthused about. Damn I wish that they had expanded on this a bit (not to start a flame war, but to give readers a fuller picture, or even to prod the FreeBSD community into "fixing" those things) edit: typo fix
- lloydatkinson 11mo agoIt does seem like a weird omission doesn’t it?
- cperciva 11mo agoOne issue, as they point out, is that we now do minor version updates every 6 months, and you need to update for each one. (We have a 3-4 month period where both are supported, but e.g. 15.0 will be EoL before 15.2 is released.) We are aware that this isn't ideal for some users, but it was a necessary tradeoff. We might be able to improve this in the future (possibly as "security updates for the base system, but no ports support") but no guarantees.
- SoftTalker 11mo agoAs noted, recent changes to OpenBSD TCP handling[1] may improve performance. On a 4 core machine I see between 12% to 22% improvement with 10 parallel TCP streams. When testing only with a single TCP stream, throughput increases between 38% to 100%. I'm not sure that directly translates to better pf performance, and four cores is hardly remarkable these days but might be typical on a small low-power router? Would be interesting if someone had a recent benchmark comparison of OpenBSD 7.8 PF vs. FreeBSD's latest. [1] https://undeadly.org/cgi?action=article;sid=20250508122430 https://undeadly.org/cgi?action=article;sid=20250508122430
- wahern 11mo agoThat particular change improves throughput received locally. Though over the past few years there's been a ton of work on unlocking the network layer generally to support more parallelism. For a firewall I guess the critical question is the degree of parallelism supported by OpenBSD's PF stack, especially as it relates to common features like connection statefulness, NAT, etc.
- SoftTalker 11mo agoThanks. Yes after I posted that I started wondering if it was really relevant to pf.
- throwaway270925 11mo agoCan confirm. Lots of performance improvements lately in OpenBSD. Our Load Balancers basically doubled throughput after updating from 7.6 to 7.7
- dylan604 11mo agoI once wrote a similar post to an DVD industry centric mailing list (remember those?) regarding switching to FCP7 from Adobe Premiere with a huge difference in how FCP7 would allow capturing of discrete audio channels vs Premiere forcing an interleaved audio stream. Eventually, a rep from Adobe contacted me through my company's PR team (a first for me) to go over the list of complaints. At the end, he agreed these were all valid complaints, and then asked "if Premiere added these changes would I be willing to switch back"? At that point, I said probably not as we'd now be fully switched to FCP7 in all departments. So I understand that sentiment as well. Honestly, I was shocked that someone actually read my missive and actually paid any mind to it. So maybe someone at OpenBSD will be as receptive if not equally unable to do anything about it.
- CodeCompost 11mo agoThe OpenBSD community does not operate like that. It's expected that you fix it yourself.
- yuvadam 11mo agoWhat's wrong with Linux for firewalls? Either openwrt, or any distro really. Why would any BSD perform better? (edit: genuinely curious why BSDs are such popular firewalls)
- 2trill2spill 11mo agoI assume in this case they already had a bunch of firewall rules for PF and switching from OpenBSD -> FreeBSD is a much easier lift then going to linux because both the BSDs are using PF, although IIRC there are some differences between both implementations.
- electric_mayhem 11mo agoPF is really nice. (Source: me. Cissp and a couple decades of professional experience with open source and proprietary firewalls). And if they are already using it on openbsd, it’s almost certainly an easier lift to move from one BSD PF implementation to another versus migrating everything to Linux and iptables.
- theideaofcoffee 11mo agoAgreed. Once you've gone pf you'll pine for it when working with anything else.
- kstrauser 11mo agoI've gotta me-too this. I've written any number of firewall rulesets on various OSes and appliances over the years, and pf is delightful. It was the first and only time I've seen a configuration file that was clearly The Way It Should Be.
- bigfatkitten 11mo agoThe only configuration language I like more is Juniper. I picked that up and became fluent in it within about a day.
- nesarkvechnep 11mo ago
- theideaofcoffee 11mo agoJust more navel-gazing from UTCC. I still don't understand why all of these submissions get upvoted so often. 10G performance just really isn't that interesting anymore, maybe around 2005 when it was the new kid on the block. If they were talking about squeezing firewall performance out of a box with a couple of 200g or 400g adapters and on run-of-the-mill CPUs and no offloading or something like Netflix publishes with their BSD work, I'd be more interested.
- cyberpunk 11mo agoIt reads a bit like someone LARP'ing a sysadmin. Perhaps they're students or something.
- yepguy 11mo agoNah, Chris is definitely a real sysadmin and his blog has been pretty popular in this space for a long time.
- kalleboo 11mo agoDespite 10G being far from new it seems like somehow they're still not even achieving that with FreeBSD. Would be more interesting to see an upgrade to Linux.
- j45 11mo agoI just like the reference to 10G ethernet. It can't become normal soon enough.
- kalleboo 11mo agoIt's ironic because I chose Linux over FreeBSD due to 10G performance. Be it a TrueNAS box with dual Xeons and Marvell 10G card or a ThinkCentre Tiny with an Intel NIC running opnSense I could never get anywhere near full 10G throughput. Switch to Linux (TrueNAS SCALE/openWrt) and it just worked at full speed. Although the article also uses weasel words like "sufficiently good" performance so it sounds like their BSD 10G performance isn't that good either.
- thatcks 11mo agoI can't remember the 10G firewall figures we got in testing off the top of my head, but we didn't max out the 10G network; I think we were getting somewhere in the 8G range. This is significantly better than our OpenBSD performance but not quite up to the level of full network speed or the full speed that two Linux machines can get talking directly to each other over our 10G network. I also suspect that performance depends on which specific NICs you have due to driver issues. The live performance of our deployed FreeBSD firewalls is harder to assess because people here don't push the network that hard very often (although every so often someone downloads a big thing from the right Internet source to get really good rates). (I'm the author of the linked-to article.)
- deleted 11mo ago[deleted]
- Y_Y 11mo agoSo you don't like OpenBSD, but you do like Ubuntu? This person seems like they know wht they are talking about and given it serious thought, but I cannot fathom how you could make such a conclusion today.
- toast0 11mo agoIf they're concerned about performance, yeah. OpenBSD doesn't do the basics that you need to get the most out of your SMP hardware; there's no way to set cpu affinity at least from userland, and it's clear that this sort of work is not a priority for OpenBSD; it's not easy work, but FreeBSD has done it. Beyond CPU affinity, you also need your network structures setup to reduce lock contention, things like fine grained locks, hashed subtables and/or "lockless" tables, configuring the NICs as close as possible to one queue per core and keeping flows on the same queue which is pinned to a single core so that the per flow locks never contend and don't bounce between cores. Ubuntu/Linux do have reasonable performance, but I think they prefer PF firewalls, so that makes Linux a non-option for firewalls. Personally, I don't really care for PF, but it offers pfsync, which I do care for, so I use it and ipfw... but I need to check in, I think FreeBSD PF may have added the hooks I use ipfw for (bandwidth limits/shaping/queue discipline).
- csmpltn 11mo ago> "there's no way to set cpu affinity at least from userland" How is that even possible. What's the excuse? On Windows, setting process affinity has been around since the Windows NT days.
- doubled112 11mo agoIt's the OS's job to manage resources.
- Twirrim 11mo agoThe OS doesn't always know everything about workloads to be able to make the right decisions.
- ThinkBeat 11mo agoI find it a bit odd that they seem to have gone from having OpenBSD as the standard and are not moving to FreeBSD and Ubuntu. I an not sure what role these computers that may transition to Ubuntu do, there are probably good reasons, I wish he had expanded on it.
- thatcks 11mo agoThe computers that moved from OpenBSD to Ubuntu were our local resolving DNS servers. These don't use PF and we also wanted to switch from our previous OpenBSD setup to Bind, where we were already running Bind on Ubuntu for our DNS master servers. The gory details were written up here: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/UsingBindNowForResolvers https://utcc.utoronto.ca/~cks/space/blog/sysadmin/UsingBindN... We may at some point switch our remaining OpenBSD DHCP server to Ubuntu (instead of to FreeBSD); like our DNS resolvers, it doesn't use PF, and we already operate a couple of Ubuntu DHCP servers. In general Ubuntu is our default choice for a Unix OS because we already run a lot of Ubuntu servers. But we have lots of PF firewall rules and no interest in trying to convert them to Linux firewall rules, so anything significant involving them is a natural environment for FreeBSD. (I'm the author of the linked-to article.)
- cyberpunk 11mo agoWhy do you say OpenBSD stopped "supporting bind"? You mean they don't include it in the base system anymore since the switch to unbound? I mean.. It's one pkg_add away. It's a weird constraint to give yourself if that was the problem, considering you absolutely had to install it on your replacement ubuntu servers.
- thatcks 11mo agoThe short version is that we wound up not feeling particularly enthused about OpenBSD itself. We have a much better developed framework for handling Ubuntu machines, making it simply easier to have some more Ubuntu machines instead of OpenBSD machines, and we also felt Bind on Ubuntu was likely to be better supported than a ports Bind on OpenBSD. If everything else is equal we're going to make a machine Ubuntu instead of OpenBSD.
- somat 11mo agoI am not very familiar with FreeBSD's pf but my understanding is that fbsd integrated it from OpenBSD and then proceeded to put a fair amount of work in making it more performant(multi core) while OpenBSD put most of it's work into improving pf's features, At this point the two pf's are different enough that they are not really compatible. OpenBSD can't really use much of fbsds multi core work and FreeBSD is A. Is a lot more hesitant about breaking backwards compatibility and B. would need get the queuing structures to work with their kernel. In short FreeBSD pf is like using an old fast version of OpenBSD pf In fact if you asked me to explain the difference between obsd and fbsd it is exactly this. fbsd focuses on performance and obsd focuses on ergonomics.
- markjdb 11mo agoThe pf maintainer in FreeBSD has been doing a ton of work to bring more recent improvements over from OpenBSD, trying to bring them in sync as much as possible without breaking compatibility: https://cgit.freebsd.org/src/log/sys/netpfil/pf https://cgit.freebsd.org/src/log/sys/netpfil/pf The state of affairs you described is much less the case now than in the past.
- popknobinfanny 11mo agoRoot on ZFS is an easy sell for me. OpenBSD's ancient filesystem is notoriously flaky, and they have no interest in replacing it anytime soon. I can't be worried that critical parts of my network won't come back up because the box spontaneously rebooted or the UPS battery ran out (yes it happens — do you load test your batteries — probably not) and their bubblegum-and-string filesystem has corruption and / and /usr won't mount and I gotta visit the console like Sam Jackson in Jurassic Park to fsck the damn thing. Firewalls are critical infra — by definition they can't be the least reliable device in the network.
- hnav 11mo agoconversely, running a firewall on something like ZFS also sounds like too much. Ideally I'd want a read-only root FS with maybe an /etc and /var managed by an overlay.
- lmm 11mo agoSounds like overcomplicating in the name of simplification. ZFS is a good, reliable, general-purpose system; often the right answer is to just put everything on ZFS and get on with your life.
- SoftTalker 11mo agoI’ve had more problems with zfs than all other filesystems combined including FAT. It’s IMO overkill for a root partition.
- denkmoon 11mo agoProblems like? I run zfs on 20gb VMs and a 100tb pool and I’ve never had a problem that wasn’t my own fault. I love root on zfs, you can snapshot your entire OS at a whim. The only other way to get that I know of is btrfs which genuinely does have well known issues.
- 11mo ago
- kuon 11mo agoI was using OpenBSD for my firewalls for a long time, but with the arrival of 10Gbit/s ethernet, I realized that I had to move back to ASIC based firewalls. Yes, you can forward 10Gbit/s with linux using VPP, but you cannot forward at that rate with small packets and stateful firewall. And it requires a lot of tuning and a large machine. A used SRX4200 from juniper runs at around 3k USD and you can even buy support for it and you can forward at like 40Gb/s IMIX with it. I still prefer PF syntax over everything else though.
- kev009 11mo agoYou can definitely build an x86 system to route 40Gb/s with small packets for under $3k and it's been the case for many years. A Xeon-D can hit 100gbps forwarding and filtering. OpenBSD is going through a slow fine grained locking transformation that FreeBSD started over 20 years ago. Eventually they will figure out they need something like epoch reclaimation, hazard pointers, or rcu.
- hapless 11mo agoI just today deployed an $800 mikrotik in my house that can route 10 gbps at wire speed. on the CPU. with firewall and nat rules applied. no joke. 4 million packets per second is, like, a lot, post-filtering and with any substantial packet size. This was doable back in 2008 with about $15k of x86 gear and a Linux kernel and a little trickery with pf_ring. The minute AMD K10 and Intel Nehalem dropped, high routing performance was mostly a software problem... Which is cool as hell, compared to the era when it required elaborate dedicated hardware, but it does not make it cheap or easy. Just, commodity. Expensive commodity. Now you can buy a device off the shelf for $800 that will do it on the CPU, to avoid the cost of Cisco or Juniper, and it has a super simple configuration interface for all the software-based features. Everything you could do in L3/L4 on a Linux platform in 2008, for like, 1/16th the price, with vastly less engineering effort. It is just like, a thing you buy, and it all kinda works outta the box. No pf_ring trickery, no deep in-house experience, just a box you buy on a web site and it moves 10 gbps with filtering for $800 There's no real magic here: they use absolutely shockingly enormous ARM chips from Amazon/Annapurna. You can build an $800 commodity platform that rivals a $15k commodity platform in 2008, and both of them replace what used to cost $500k. Is it as good as Cisco or Juniper? oh, certainly not. Will it route and filter traffic at much greater rates, for $800, than anything they have ever been bothered to offer? ABSOLUTELY
- asteroidburger 11mo agoWho is the "we" here? I've poked around a few pages on this fellow's site, and apparently haven't found the right one to answer that.
- somat 11mo agoWhere he works... Or perhaps more accurately his co-workers. which is the domain. The university of Toronto.
- hapless 11mo agoThis post sounded faintly crazy to me, so I went into a little wiki-hole consisting primarily of mailing lists and dev docs Turns out, the main reason `pf` is non-portable is that half of it runs inside Berkeley-type network stacks, often in kernel space, but the remainder is in user space. So the miserable single-threaded `pf` on OpenBSD is still, in some part, single-threaded on FreeBSD, but for certain rule-sets, you will get the benefits of FreeBSD's intensively re-entrant and multithreaded TCP/IP, because those parts of `pf` are embedded in the network stack. So depending on workload, a given `pf` configuration on OpenBSD might be perfectly equal to its FreeBSD counterpart, or hundreds of times slower. I feel like this gives a lot of context to the OP's grousing around "10 gbps" P.S. To confess my own biases: a port of a `pf` configuration to a platform where some rulesets are high performance and others are not, that would not be very attractive to me. An improvement, but not a solution. I would be looking to move to a Linux stack. Baby steps, I guess. I have done worse things to better people! P.P.S. I suspect this coupling between a re-entrant TCP/IP stack and a single-threaded firewall process is also why FreeBSD `pf` is never even close to feature parity with its OpenBSD counterpart -- it is just easier to do new stuff with a simpler model