7 ms·

OpenBSD PF versus FreeBSD PF

198 points by zdw 3y ago

tiffanyh 3y ago
Don't forget DragonflyBSD version of PF. They look an old version of OpenBSD PF and updated it to be multithreaded. I could be mistaken, but I believe both FreeBSD PF & OpenBSD PF are still single threaded. https://man.dragonflybsd.org/?command=pf&section=4 https://man.dragonflybsd.org/?command=pf&section=4
- postmodest 3y ago
  The only advantage I can see to MT PF would be multiple NICs. And even then they have to be really beefy NICs
  - smashed 3y ago
    From what I understand, modern NICs have multiple Rx/tx queues and thus the work of sending and receiving can be assigned to different kernel threads. The number of queues will vary according to the specific NIC chip.
    - toast0 3y ago
      I was going to say something similar. Multiqueue NICs do great work on networking loads when you have 1 core per NIC queue; and you can eliminate or at least reduce cross-core communication for most of the work. It's not complex firewalling, but I did some HAProxy stuff in tcp mode, and the throughput available when running without queue alignment was miniscule compared to the throughput available when properly aligned. Firewalling has the benefit that queue alignment should happen automagically, because of where it runs. If you're doing a lot of processing in userspace, it makes a lot less of a difference, but on a very lightweight application, there was no point in using more cores than nic queues, because cross-core communication was too slow.
- darkhelmet 3y ago
  DragonFly and FreeBSD have radically different ideas on how multithreading should work. (Understatement of the century right there!) FWIW the threading in the network stack is one of the original major divergences between Open/FreeBSD PF. The way FreeBSD's PF works is within the context of FreeBSD's threaded network stack. FreeBSD PF is not "single threaded" in the way it used to mean. I would really like to have the modern PF syntax/structure from OpenBSD in FreeBSD though. The unified rdr/nat/pass/block mechanism in OpenBSD is so much cleaner and nicer to work with.
  - cperciva 3y ago
    DragonFly and FreeBSD have radically different ideas on how multithreading should work. (Understatement of the century right there!) To elaborate for people who don't know the history: Disagreement about how multithreading should work are why Matt got booted from the FreeBSD project and started DragonFly.
    - darkhelmet 3y ago
      It's not that simple. Disagreements are one thing, but the real reason was the conflict and drama that happened when things didn't go his way. The threading approach just happened to be a really good trigger for those conflicts. Conflicting personalities makes resolving technical disagreements so much harder.
      - cperciva 3y ago
        I was being euphemistic. It was a "doesn't play well with others" issue, but it came to the fore with SMP.
        bch 3y ago
        Playing well obviously important (and I wasn’t involved, have no other insight), but do you know if the poor showing for fbsd 5.x correspond w the disagreements?
- dmm 3y ago
  FreeBSD pf has been multithreaded for a long time. The FreeBSD 11 pf(4) man page from 2013 specifically says so[1]. [1] https://man.freebsd.org/cgi/man.cgi?query=pf&apropos=0&sektion=4&manpath=FreeBSD+11.0-RELEASE&arch=default&format=html#end https://man.freebsd.org/cgi/man.cgi?query=pf&apropos=0&sekti...
  - Lammy 3y ago
    "SMP" might be a more useful search term. You're correct on the timeframe — merged into HEAD in 2012 https://lists.freebsd.org/pipermail/freebsd-pf/2012-September/006740.html https://lists.freebsd.org/pipermail/freebsd-pf/2012-Septembe... I'm not an OpenBSD user but it sounds like this changed recently in OpenBSD-land as well, because the "Will multiple processors help?" FAQ entry disappeared some time between January and March 2023: https://web.archive.org/web/20230112170731/https://www.openbsd.org/faq/pf/perf.html https://web.archive.org/web/20230112170731/https://www.openb...
    - gigatexal 3y ago
      I didn’t know this but it makes sense. Either way for most of the use cases (homelabs on residential internet speeds) either FBSD or OpenBSD’s PF will perform just swimmingly.
josteink 3y ago
> And the PF syntax is the most approachable in all of open source Unix. As someone not deeply into the BSDs and who has never tried pf, I would love to see this claim illustrated with a few examples. Are there any “definitive” or authoritative beginner guides for those merely curious, but without a production system to test on?
- i_am_a_peasant 3y ago
  I'd go for the handbook in that case: https://docs.freebsd.org/en/books/handbook/firewalls/#firewalls-pf https://docs.freebsd.org/en/books/handbook/firewalls/#firewa...
- nesarkvechnep 3y ago
  Get yourself “The Book of PF” and get to work.
- tedunangst 3y ago
  Find your favorite topic in https://www.openbsd.org/faq/pf/index.html https://www.openbsd.org/faq/pf/index.html and compare to how you'd do it in your favorite firewall.
  - lacrosse_tannin 3y ago
    How do you do endpoint independent nat?
- voytec 3y ago
  > And the PF syntax is the most approachable in all of open source Unix. I actually find ipfw(8) [0] syntax more human-friendly. [0] https://man.freebsd.org/cgi/man.cgi?query=ipfw&sektion=8&format=html https://man.freebsd.org/cgi/man.cgi?query=ipfw&sektion=8&for...
  - stock_toaster 3y ago
    I wish dragonflyBSD’s ipfw3 would get ported to freeBSD.
jayp1418 3y ago
Also don't forget NetBSD 's npf
- idatum 3y ago
  I wanted to use npf but I couldn't figure out how to implement NAT64. The syntax looked nice though.
  - systems_glitch 3y ago
    Juuuust different enough to be aggravating :P Check the /etc/examples stuff for the best docs on `npf`. Most of what I found online was outdated or just plain wrong.
drewg123 3y ago
And if you're using FreeBSD with a fast NIC, look into the pfilctl command and use a "head" input hook into your NIC driver's ethernet input handling routine. Doing this allows the firewall input checks to happen early, before the packet has been received into the network stack, while its still in DMA-mapped, driver-owned buffers. If the firewall decides a packet is to be dropped, the NIC can recycle the buffer with minimal overhead (skips DMA-unmap, buffer free, ethernet input into the IP stack, and the replacement buffer alloc, DMA map). This allows for very fast dropping of DOS flood attacks. I've tested this using ipfw up to screening and dropping several 10s of millions of packets on 2014 era Xeons with minimal impact to traffic serving. I wrote a paper about this for a conf. that was cancelled due to COVID. I really need to re-submit it someplace. This works for Mellanox, Chelsio, and NICs that use iflib (ix, ixl, igb, bnxt, and more that I'm forgetting).
- digitalsushi 3y ago
  I'm picturing that scene a few episodes into the soul art anime - I barely remember it but the main character was so overpowered, he was standing there absorbing a full barrage from a much lower ranked opponent flooding him with punches and kicks; the main character's health regenerated so quickly that the attack didn't have effects. In 2014 I was working on a hardware appliance for a company that has something to do with packet capture, and I found an intel driver that implemented a ring buffer on a 1 gigabit Ethernet adapter that allowed me to capture line rate without dropping a single packet over the course of hours; prior to this the adapter was barely able to capture 90% of the packets. I remember marveling at the design that must have gone into it, and here too to your description of this performance improvement. I miss that stuff. Thanks for sharing,
- jpk 3y ago
  Would love to watch that talk (so please re-submit it somewhere!), but in lieu of that, is the paper available anywhere?
- bravetraveler 3y ago
  This is awesome to see [as a mostly-Linux person] - people have managed to impress the sentiment on me that the BSDs are well-suited for networking. I never quite 'got it' - this helps. Are you aware of similar tweaks in Linux land, by chance? I consider myself reasonably well-experienced as a systems administrator... but can't quite recall stumbling across anything quite like this.
wkat4242 3y ago
And then there's macOS pf too. Also different again!
- cyberpunk 3y ago
  I'm sure i read on a comment here recently that the latest macos is using a recent openbsd pf? As quite a heavy user of freebsd pf, I wonder if anyone knows more details on that?
  - nvy 3y ago
    On my mbp, the man page for `pfctl` contains the following: >HISTORY > The pfctl program and the packet filter mechanism first appeared in OpenBSD 3.0.
    - yakubin 3y ago
      The same is in the current manpage on OpenBSD: <https://man.openbsd.org/pfctl https://man.openbsd.org/pfctl>. The next line is more telling though: > July 1, 2007 Although I'm running Monterey, so maybe it's updated in more recent versions of MacOS.
      - nvy 3y ago
        Mine doesn't have a date at the bottom, just a macOS version number.
        nvy 3y ago
        edit: never mind, it does
        wkat4242 3y ago
        Still 2007 here in macOS Sonoma 14.0 I'm just super surprised it's there at all considering no rules are defined by the OS, and nobody uses it anymore because all the firewall vendors moved over to system extensions.
Anya200 3y ago
[flagged]
tedunangst 3y ago
For a bit of context regarding some of the "unfixed" issues in FreeBSD, some of it comes down to different security contexts. In openbsd, pf is pretty much root only, and root -> kernel escalation isn't a big deal. There's a bug, it gets fixed, but no hoopla. Easy to miss if you're not paying close attention. Maybe fixed incidentally as part of a refactor. But FreeBSD also has jails/vnet, which makes root -> kernel escalation a lot more spicy. Eventually the bug gets found, some years later, although not really the result of negligence.
manifoldgeo 3y ago
Michael W. Lucas is such a good author, and I just wanted to give a shout-out: "TLS Mastery" opened my eyes to the world of TLS / SSL certs in an understandable way that used interactive examples on Linux.
paradox242 3y ago
That felt like it was over before it really got started but I guess that was the point.

198 points by zdw 3y ago

Don't forget DragonflyBSD version of PF. They look an old version of OpenBSD PF and updated it to be multithreaded. I could be mistaken, but I believe both FreeBSD PF & OpenBSD PF are still single threaded. https://man.dragonflybsd.org/?command=pf&section=4 https://man.dragonflybsd.org/?command=pf&section=4

postmodest 3y ago
The only advantage I can see to MT PF would be multiple NICs. And even then they have to be really beefy NICs
- smashed 3y ago
  From what I understand, modern NICs have multiple Rx/tx queues and thus the work of sending and receiving can be assigned to different kernel threads. The number of queues will vary according to the specific NIC chip.
  - toast0 3y ago
    I was going to say something similar. Multiqueue NICs do great work on networking loads when you have 1 core per NIC queue; and you can eliminate or at least reduce cross-core communication for most of the work. It's not complex firewalling, but I did some HAProxy stuff in tcp mode, and the throughput available when running without queue alignment was miniscule compared to the throughput available when properly aligned. Firewalling has the benefit that queue alignment should happen automagically, because of where it runs. If you're doing a lot of processing in userspace, it makes a lot less of a difference, but on a very lightweight application, there was no point in using more cores than nic queues, because cross-core communication was too slow.
darkhelmet 3y ago
DragonFly and FreeBSD have radically different ideas on how multithreading should work. (Understatement of the century right there!) FWIW the threading in the network stack is one of the original major divergences between Open/FreeBSD PF. The way FreeBSD's PF works is within the context of FreeBSD's threaded network stack. FreeBSD PF is not "single threaded" in the way it used to mean. I would really like to have the modern PF syntax/structure from OpenBSD in FreeBSD though. The unified rdr/nat/pass/block mechanism in OpenBSD is so much cleaner and nicer to work with.
- cperciva 3y ago
  DragonFly and FreeBSD have radically different ideas on how multithreading should work. (Understatement of the century right there!) To elaborate for people who don't know the history: Disagreement about how multithreading should work are why Matt got booted from the FreeBSD project and started DragonFly.
  - darkhelmet 3y ago
    It's not that simple. Disagreements are one thing, but the real reason was the conflict and drama that happened when things didn't go his way. The threading approach just happened to be a really good trigger for those conflicts. Conflicting personalities makes resolving technical disagreements so much harder.
    - cperciva 3y ago
      I was being euphemistic. It was a "doesn't play well with others" issue, but it came to the fore with SMP.
      - bch 3y ago
        Playing well obviously important (and I wasn’t involved, have no other insight), but do you know if the poor showing for fbsd 5.x correspond w the disagreements?
dmm 3y ago
FreeBSD pf has been multithreaded for a long time. The FreeBSD 11 pf(4) man page from 2013 specifically says so[1]. [1] https://man.freebsd.org/cgi/man.cgi?query=pf&apropos=0&sektion=4&manpath=FreeBSD+11.0-RELEASE&arch=default&format=html#end https://man.freebsd.org/cgi/man.cgi?query=pf&apropos=0&sekti...
- Lammy 3y ago
  "SMP" might be a more useful search term. You're correct on the timeframe — merged into HEAD in 2012 https://lists.freebsd.org/pipermail/freebsd-pf/2012-September/006740.html https://lists.freebsd.org/pipermail/freebsd-pf/2012-Septembe... I'm not an OpenBSD user but it sounds like this changed recently in OpenBSD-land as well, because the "Will multiple processors help?" FAQ entry disappeared some time between January and March 2023: https://web.archive.org/web/20230112170731/https://www.openbsd.org/faq/pf/perf.html https://web.archive.org/web/20230112170731/https://www.openb...
  - gigatexal 3y ago
    I didn’t know this but it makes sense. Either way for most of the use cases (homelabs on residential internet speeds) either FBSD or OpenBSD’s PF will perform just swimmingly.

josteink 3y ago

> And the PF syntax is the most approachable in all of open source Unix. As someone not deeply into the BSDs and who has never tried pf, I would love to see this claim illustrated with a few examples. Are there any “definitive” or authoritative beginner guides for those merely curious, but without a production system to test on?

i_am_a_peasant 3y ago
I'd go for the handbook in that case: https://docs.freebsd.org/en/books/handbook/firewalls/#firewalls-pf https://docs.freebsd.org/en/books/handbook/firewalls/#firewa...
nesarkvechnep 3y ago
Get yourself “The Book of PF” and get to work.
tedunangst 3y ago
Find your favorite topic in https://www.openbsd.org/faq/pf/index.html https://www.openbsd.org/faq/pf/index.html and compare to how you'd do it in your favorite firewall.
- lacrosse_tannin 3y ago
  How do you do endpoint independent nat?
voytec 3y ago
> And the PF syntax is the most approachable in all of open source Unix. I actually find ipfw(8) [0] syntax more human-friendly. [0] https://man.freebsd.org/cgi/man.cgi?query=ipfw&sektion=8&format=html https://man.freebsd.org/cgi/man.cgi?query=ipfw&sektion=8&for...
- stock_toaster 3y ago
  I wish dragonflyBSD’s ipfw3 would get ported to freeBSD.

jayp1418 3y ago

Also don't forget NetBSD 's npf

idatum 3y ago
I wanted to use npf but I couldn't figure out how to implement NAT64. The syntax looked nice though.
- systems_glitch 3y ago
  Juuuust different enough to be aggravating :P Check the /etc/examples stuff for the best docs on `npf`. Most of what I found online was outdated or just plain wrong.

drewg123 3y ago

And if you're using FreeBSD with a fast NIC, look into the pfilctl command and use a "head" input hook into your NIC driver's ethernet input handling routine. Doing this allows the firewall input checks to happen early, before the packet has been received into the network stack, while its still in DMA-mapped, driver-owned buffers. If the firewall decides a packet is to be dropped, the NIC can recycle the buffer with minimal overhead (skips DMA-unmap, buffer free, ethernet input into the IP stack, and the replacement buffer alloc, DMA map). This allows for very fast dropping of DOS flood attacks. I've tested this using ipfw up to screening and dropping several 10s of millions of packets on 2014 era Xeons with minimal impact to traffic serving. I wrote a paper about this for a conf. that was cancelled due to COVID. I really need to re-submit it someplace. This works for Mellanox, Chelsio, and NICs that use iflib (ix, ixl, igb, bnxt, and more that I'm forgetting).

digitalsushi 3y ago
I'm picturing that scene a few episodes into the soul art anime - I barely remember it but the main character was so overpowered, he was standing there absorbing a full barrage from a much lower ranked opponent flooding him with punches and kicks; the main character's health regenerated so quickly that the attack didn't have effects. In 2014 I was working on a hardware appliance for a company that has something to do with packet capture, and I found an intel driver that implemented a ring buffer on a 1 gigabit Ethernet adapter that allowed me to capture line rate without dropping a single packet over the course of hours; prior to this the adapter was barely able to capture 90% of the packets. I remember marveling at the design that must have gone into it, and here too to your description of this performance improvement. I miss that stuff. Thanks for sharing,
jpk 3y ago
Would love to watch that talk (so please re-submit it somewhere!), but in lieu of that, is the paper available anywhere?
bravetraveler 3y ago
This is awesome to see [as a mostly-Linux person] - people have managed to impress the sentiment on me that the BSDs are well-suited for networking. I never quite 'got it' - this helps. Are you aware of similar tweaks in Linux land, by chance? I consider myself reasonably well-experienced as a systems administrator... but can't quite recall stumbling across anything quite like this.

wkat4242 3y ago

And then there's macOS pf too. Also different again!

cyberpunk 3y ago
I'm sure i read on a comment here recently that the latest macos is using a recent openbsd pf? As quite a heavy user of freebsd pf, I wonder if anyone knows more details on that?
- nvy 3y ago
  On my mbp, the man page for `pfctl` contains the following: >HISTORY > The pfctl program and the packet filter mechanism first appeared in OpenBSD 3.0.
  - yakubin 3y ago
    The same is in the current manpage on OpenBSD: <https://man.openbsd.org/pfctl https://man.openbsd.org/pfctl>. The next line is more telling though: > July 1, 2007 Although I'm running Monterey, so maybe it's updated in more recent versions of MacOS.
    - nvy 3y ago
      Mine doesn't have a date at the bottom, just a macOS version number.
      - nvy 3y ago
        edit: never mind, it does
      - wkat4242 3y ago
        Still 2007 here in macOS Sonoma 14.0 I'm just super surprised it's there at all considering no rules are defined by the OS, and nobody uses it anymore because all the firewall vendors moved over to system extensions.

Anya200 3y ago

[flagged]

tedunangst 3y ago

For a bit of context regarding some of the "unfixed" issues in FreeBSD, some of it comes down to different security contexts. In openbsd, pf is pretty much root only, and root -> kernel escalation isn't a big deal. There's a bug, it gets fixed, but no hoopla. Easy to miss if you're not paying close attention. Maybe fixed incidentally as part of a refactor. But FreeBSD also has jails/vnet, which makes root -> kernel escalation a lot more spicy. Eventually the bug gets found, some years later, although not really the result of negligence.

manifoldgeo 3y ago

Michael W. Lucas is such a good author, and I just wanted to give a shout-out: "TLS Mastery" opened my eyes to the world of TLS / SSL certs in an understandable way that used interactive examples on Linux.

paradox242 3y ago

That felt like it was over before it really got started but I guess that was the point.