7 ms·

In what sometimes feels like a Linux world, it is good to be reminded of some of the excellent quality projects and codebases in the BSD projects. We still have

In what sometimes feels like a Linux world, it is good to be reminded of some of the excellent quality projects and codebases in the BSD projects. We still have a few Open and Free BSD servers running, as they are basically rock solid, and have a much slower rate of evolution, and decent stability, unlike some of the Linux distros we run, where constant rejiggling of the underpinnings, init system etc, were a bit painful.

SEJeff 12y ago
Facebook uses Linux "where it is faster than FreeBSD, which is basically everywhere" per a recent whitepaper or presentation I saw. What I found interesting, is that Facebook runs FreeBSD in production at all. It just goes to show how solid the BSDs really are. I use the hell out of keepalived (OSS vrrp daemon) + conntrackd on Linux, but strongly prefer carpd and pf. It moves a lot slower, but it caters to a different audience.
- krakensden 12y ago
  whatsapp was a FreeBSD shop, and they weren't purchased that long ago.
  - X-Istence 12y ago
    WhatsApp is still a FreeBSD shop AFAIK.
- sturadnidge 12y ago
  Be interested to see that paper / presentation if you can track it down, especially given https://www.facebook.com/careers/department?req=a0IA000000Cz53VMAR https://www.facebook.com/careers/department?req=a0IA000000Cz...
- justincormack 12y ago
  They have a job ad out [1] saying "Facebook is seeking a Linux Kernel Software Engineer to join our Kernel team, with a primary focus on the networking subsystem. Our goal over the next few years is for the Linux kernel network stack to rival or exceed that of FreeBSD." [1] https://www.facebook.com/careers/department?req=a0IA000000Cz53VMAR https://www.facebook.com/careers/department?req=a0IA000000Cz...
  - contingencies 12y ago
    That's very interesting, but isn't very convincing. Is there some kind of study on the supposed differences? I like BSD, but I've always felt from admin experience that the Linux networking stack is far more capable in terms of features.
    - UNIXgod 12y ago
      BSD is the implementation to the networking stack.
      - contingencies 12y ago
        From the API design / pedigree side, sure. But that's a given, and a tangent to the curiosity I am trying to have sated :)
    - justincormack 12y ago
      Well obviously it is a brief statement. Linux generally does have enormous numbers of features, but they are no use if they do not apply to you. My guesses about Facebook is that network stack issues would be around ipv6, performance or large scale (eg lost of routes).
- Keyframe 12y ago
  Define faster though.
  - SEJeff 12y ago
    There are 2 common things where "faster" makes sense. 1. Throughput 2. Latency Both are relevant for webapps like facebook.
- allegory 12y ago
  An anecdote. When the company I was working for at the time's AC failed in the machine room and melted three racks of kit, all 7 of our FreeBSD machines at the time were back online from scratch on hot spare kit in under two hours. Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place. Quality engineering, documentation and automation ability shits on everything else IMHO and that's where FreeBSD lands every time. Performance isn't a major thing for me. Linux on the other hand is heading in the direction of windows. I'll probably get flamed by the systemd proponents here but it took me three hours to work out how to debug this command: timedatectl set-ntp yes Which returned: Failed to issue message call
  - incision 12y ago
    >'Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place.' I'd be overjoyed if I never had to admin Windows again, but it's a bit unfair to frame that as an OS comparison. The most basic of direct from TechNet 'best practices' would reduce the impact of a failure of that sort to minutes. That said, that sort of failure not at all uncommon. Further, I expect such disasters would happen far less often if an existing, poorly implemented Microsoft environment could be remediated into something better as directly as anything unix-like.
    - allegory 12y ago
      I think it's completely fair. Windows is completely non-deterministic due to the insane amount of coupling inside it and the sheer snowball of crap that has stuck to it over the years. In this case, they had tested DR strategy etc in place. Unfortunately when it came to doing it for real (onto the same backup hardware the DR plan was tested against), TSHTF and Windows threw an obscure COM error loading the AD catalog. That required contacting MS 1st line who didn't know what it was so had to escalate it to the AD guys. One obscure registry key change and an ACL change (related to ESE) and it was back. All best practices followed, yet complexity, poor design and secret knowledge crept in and shot the whole process. Stuff like that scares the shit out of me having been on the end of it way too many times now. Many a night have I spent up at 2AM trying to work out why the hell something odd has gone on inside Windows and taken something out in production. Not once in 20 years has a proper Unix (Solaris/FreeBSD) box woken me up or shafted me for hours.
- jedberg 12y ago
  reddit originally ran on FreeBSD. They switched to Linux before I got there for ease of administration. I wanted to switch it back to BSD, but never got around to it as it wasn't very high priority. I tried to switch again when we moved to Amazon, but sadly it wasn't an option so I went to Ubuntu instead. I've always thought that BSD is superior from an admin perspective, as in it's easier to do it right and keep it solid. I may be biased though as my first job was IT for a company that had BSD committers so everything we did was on BSD and I learned how to admin from arguably some of the best BSD admins in the world.
  - lifeisstillgood 12y ago
    I would be interested in interviewing (for the worlds smallest blog audience) some of "the worlds best BSD admins" - is there any chance of an introduction ?
- ams6110 12y ago
  BSDs do change stuff too. OpenBSD recently replaced sendmail with OpenSMTPD, and apache with nginx. FreeBSD 10 has some big changes[1], depending on what you're doing with it. Still, in general I agree. I run OpenBSD as my main desktop system because it works and mostly stays out of my way. [1] http://www.freebsd.org/releases/10.0R/announce.html http://www.freebsd.org/releases/10.0R/announce.html
  - Forbo 12y ago
    Just a heads-up, OpenBSD is actually removing nginx from the base install in favor of httpd. http://undeadly.org/cgi?action=article&sid=20140827065755 http://undeadly.org/cgi?action=article&sid=20140827065755 Edit: Adding in the link to the actual commit: http://marc.info/?l=openbsd-cvs&m=140908174910713&w=2 http://marc.info/?l=openbsd-cvs&m=140908174910713&w=2
    - ams6110 12y ago
      Wow that was pretty fast, by OpenBSD standards. nginx was just added to base a release or two ago. Thanks.
      - talideon 12y ago
        It's understandable when you know that httpd was spun out of relayd: they basically had almost all the work done and figured they'd might as well do the last few bits needed ot make it a standalone HTTP daemon.
  - cjg_ 12y ago
    Actually, OpenBSD just replaced nginx with their own httpd.
    - Nux 12y ago
      What httpd would that be? I thought they replaced their custom Apache with (also custom) Nginx.
squiguy7 12y ago
Not to mention, you can run this on Linux too so you are not limited to using BSD considering it is a type 2 hypervisor.
- justincormack 12y ago
  You can run Linux as a host on bhyve, but bhyve has not been ported to run on a Linux host yet.
  - squiguy7 12y ago
    Oh shoot, read that wrong!
XERQ 12y ago
I've used FreeBSD since the 6.0 days, and agree that it's absolutely rock solid. The main problem is that the support cycles are way too short, for instance 10.0 is EOL in 4 months (1 year after it was released), whereas RHEL is EOL in 10 years.
- _delirium 12y ago
  I think some of the reason for that is that enterprise usage of FreeBSD is frequently assumed to be via an intermediary, not directly as an actual FreeBSD install. RHEL directly supports enterprise installs, but FreeBSD is doing something more like producing a reference implementation. The enterprise installs that need N years of support lifecycle are supposed to run on a derived system (one of NetApp's, say), with its own EOL policies.
- X-Istence 12y ago
  This is being worked on, in that 10.x is going to be supported for at least 2 years, and in the future there has been talks about 5 years of support per major release. The minor releases will get dropped sooner, upgrading is mostly painless anyway (from 9.1 to 9.2 was very simple) so it shouldn't make much of a difference.
- justincormack 12y ago
  There is now going to be commercial support for 15 years. There was a discussion with the company doing it on bsdnow.tv recently.
- jrapdx3 12y ago
  As others have commented, the effective EOL is not really that short. FBSD 10.0 was released Jan 2014, so 10.0 will be supported for a year or so. Updating to 10.1 (when it is finalized in Q4) will very likely be painless, no incompatibilities, so as a practical matter, EOL is extended at least another year or so. This has been the history of FBSD since my exposure to 4.x back in 2000. A more recent example, FBSD 8.0 was announced Nov 2009. And now, nearly 5 years later, FBSD 8.4 is still a supported version. Sure, technically each minor version of 8.x reached EOL after a year. But since keeping an 8.x installation up-to-date has been so seamless, it achieves an EOL >=5 years from introduction. Anyway, how I see it based on real-world experience.
donavanm 12y ago
One aspect thats always stood at is the system/userland divide. The BSDs have both kernel and userland developed by the same folks. The leads to a consistent, enjoyable, user experience. In linux land the divide between kernel and user is quite stark. Throwing problems overthe fence, and blaming the "other side", is far too common. The state of documentation and man pages is indicative. Nearly everything in the BSDs has a well written up to date docs. Determining behavior in linux all too often comes down to reading opaque kernel code and old mailing list threads that may or may not be relevant.
- _delirium 12y ago
  That's the standard line, but I have never felt that as a user. Debian and FreeBSD feel about equally integrated to me as systems. Some stuff in both is developed by different upstreams (large parts of both the Debian and FreeBSD base installs are developed elsewhere), but the "buck stops here" when it comes to integration, integration-testing, engineering a consistent user experience, and generally making the system work for end-users. If anything Debian feels slightly more tightly integrated to me, though it depends on whether you frequently use a lot of non-base packages. FreeBSD's model is tight integration of the base install and light curation of everything else in userland, while Debian's model is to "Debianize" everything in the archive and ensure it plays well together, even non-base packages.
pjmlp 12y ago
The worse is that many now seem to equate GNU/Linux to UNIX, thus forgetting not all UNIX are alike, nor POSIX is a portable as it seems, X is not everywhere and many other differences.
- clarry 12y ago
  Another thing that bothers me a lot is that people equate GNU or Linux or GNU/Linux to free software.
- _delirium 12y ago
  I think market share has mostly led to that. As long as the commercial unixes had significant market share (esp. among paying customers), people developing Unix software had an incentive to write it portably, so it could run on at least Solaris and Linux. As the bottom dropped out of the Solaris market, a lot of packages decided not to bother, as >90% of Unix customers were on Linux. Some don't even bother supporting more than one specific Linux distribution (among commercial software packages, that's often RHEL, sometimes RHEL+Ubuntu). There was a period when non-x86 compatibility went through a similar low period (due to the decline of Alpha, SPARC, etc.), but increasing ARM market share is making architecture portability more practically relevant again.

SEJeff 12y ago

Facebook uses Linux "where it is faster than FreeBSD, which is basically everywhere" per a recent whitepaper or presentation I saw. What I found interesting, is that Facebook runs FreeBSD in production at all. It just goes to show how solid the BSDs really are. I use the hell out of keepalived (OSS vrrp daemon) + conntrackd on Linux, but strongly prefer carpd and pf. It moves a lot slower, but it caters to a different audience.

krakensden 12y ago
whatsapp was a FreeBSD shop, and they weren't purchased that long ago.
- X-Istence 12y ago
  WhatsApp is still a FreeBSD shop AFAIK.
sturadnidge 12y ago
Be interested to see that paper / presentation if you can track it down, especially given https://www.facebook.com/careers/department?req=a0IA000000Cz53VMAR https://www.facebook.com/careers/department?req=a0IA000000Cz...
justincormack 12y ago
They have a job ad out [1] saying "Facebook is seeking a Linux Kernel Software Engineer to join our Kernel team, with a primary focus on the networking subsystem. Our goal over the next few years is for the Linux kernel network stack to rival or exceed that of FreeBSD." [1] https://www.facebook.com/careers/department?req=a0IA000000Cz53VMAR https://www.facebook.com/careers/department?req=a0IA000000Cz...
- contingencies 12y ago
  That's very interesting, but isn't very convincing. Is there some kind of study on the supposed differences? I like BSD, but I've always felt from admin experience that the Linux networking stack is far more capable in terms of features.
  - UNIXgod 12y ago
    BSD is the implementation to the networking stack.
    - contingencies 12y ago
      From the API design / pedigree side, sure. But that's a given, and a tangent to the curiosity I am trying to have sated :)
  - justincormack 12y ago
    Well obviously it is a brief statement. Linux generally does have enormous numbers of features, but they are no use if they do not apply to you. My guesses about Facebook is that network stack issues would be around ipv6, performance or large scale (eg lost of routes).
Keyframe 12y ago
Define faster though.
- SEJeff 12y ago
  There are 2 common things where "faster" makes sense. 1. Throughput 2. Latency Both are relevant for webapps like facebook.
allegory 12y ago
An anecdote. When the company I was working for at the time's AC failed in the machine room and melted three racks of kit, all 7 of our FreeBSD machines at the time were back online from scratch on hot spare kit in under two hours. Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place. Quality engineering, documentation and automation ability shits on everything else IMHO and that's where FreeBSD lands every time. Performance isn't a major thing for me. Linux on the other hand is heading in the direction of windows. I'll probably get flamed by the systemd proponents here but it took me three hours to work out how to debug this command: timedatectl set-ntp yes Which returned: Failed to issue message call
- incision 12y ago
  >'Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place.' I'd be overjoyed if I never had to admin Windows again, but it's a bit unfair to frame that as an OS comparison. The most basic of direct from TechNet 'best practices' would reduce the impact of a failure of that sort to minutes. That said, that sort of failure not at all uncommon. Further, I expect such disasters would happen far less often if an existing, poorly implemented Microsoft environment could be remediated into something better as directly as anything unix-like.
  - allegory 12y ago
    I think it's completely fair. Windows is completely non-deterministic due to the insane amount of coupling inside it and the sheer snowball of crap that has stuck to it over the years. In this case, they had tested DR strategy etc in place. Unfortunately when it came to doing it for real (onto the same backup hardware the DR plan was tested against), TSHTF and Windows threw an obscure COM error loading the AD catalog. That required contacting MS 1st line who didn't know what it was so had to escalate it to the AD guys. One obscure registry key change and an ACL change (related to ESE) and it was back. All best practices followed, yet complexity, poor design and secret knowledge crept in and shot the whole process. Stuff like that scares the shit out of me having been on the end of it way too many times now. Many a night have I spent up at 2AM trying to work out why the hell something odd has gone on inside Windows and taken something out in production. Not once in 20 years has a proper Unix (Solaris/FreeBSD) box woken me up or shafted me for hours.
jedberg 12y ago
reddit originally ran on FreeBSD. They switched to Linux before I got there for ease of administration. I wanted to switch it back to BSD, but never got around to it as it wasn't very high priority. I tried to switch again when we moved to Amazon, but sadly it wasn't an option so I went to Ubuntu instead. I've always thought that BSD is superior from an admin perspective, as in it's easier to do it right and keep it solid. I may be biased though as my first job was IT for a company that had BSD committers so everything we did was on BSD and I learned how to admin from arguably some of the best BSD admins in the world.
- lifeisstillgood 12y ago
  I would be interested in interviewing (for the worlds smallest blog audience) some of "the worlds best BSD admins" - is there any chance of an introduction ?
ams6110 12y ago
BSDs do change stuff too. OpenBSD recently replaced sendmail with OpenSMTPD, and apache with nginx. FreeBSD 10 has some big changes[1], depending on what you're doing with it. Still, in general I agree. I run OpenBSD as my main desktop system because it works and mostly stays out of my way. [1] http://www.freebsd.org/releases/10.0R/announce.html http://www.freebsd.org/releases/10.0R/announce.html
- Forbo 12y ago
  Just a heads-up, OpenBSD is actually removing nginx from the base install in favor of httpd. http://undeadly.org/cgi?action=article&sid=20140827065755 http://undeadly.org/cgi?action=article&sid=20140827065755 Edit: Adding in the link to the actual commit: http://marc.info/?l=openbsd-cvs&m=140908174910713&w=2 http://marc.info/?l=openbsd-cvs&m=140908174910713&w=2
  - ams6110 12y ago
    Wow that was pretty fast, by OpenBSD standards. nginx was just added to base a release or two ago. Thanks.
    - talideon 12y ago
      It's understandable when you know that httpd was spun out of relayd: they basically had almost all the work done and figured they'd might as well do the last few bits needed ot make it a standalone HTTP daemon.
- cjg_ 12y ago
  Actually, OpenBSD just replaced nginx with their own httpd.
  - Nux 12y ago
    What httpd would that be? I thought they replaced their custom Apache with (also custom) Nginx.

squiguy7 12y ago

Not to mention, you can run this on Linux too so you are not limited to using BSD considering it is a type 2 hypervisor.

justincormack 12y ago
You can run Linux as a host on bhyve, but bhyve has not been ported to run on a Linux host yet.
- squiguy7 12y ago
  Oh shoot, read that wrong!

XERQ 12y ago

I've used FreeBSD since the 6.0 days, and agree that it's absolutely rock solid. The main problem is that the support cycles are way too short, for instance 10.0 is EOL in 4 months (1 year after it was released), whereas RHEL is EOL in 10 years.

_delirium 12y ago
I think some of the reason for that is that enterprise usage of FreeBSD is frequently assumed to be via an intermediary, not directly as an actual FreeBSD install. RHEL directly supports enterprise installs, but FreeBSD is doing something more like producing a reference implementation. The enterprise installs that need N years of support lifecycle are supposed to run on a derived system (one of NetApp's, say), with its own EOL policies.
X-Istence 12y ago
This is being worked on, in that 10.x is going to be supported for at least 2 years, and in the future there has been talks about 5 years of support per major release. The minor releases will get dropped sooner, upgrading is mostly painless anyway (from 9.1 to 9.2 was very simple) so it shouldn't make much of a difference.
justincormack 12y ago
There is now going to be commercial support for 15 years. There was a discussion with the company doing it on bsdnow.tv recently.
jrapdx3 12y ago
As others have commented, the effective EOL is not really that short. FBSD 10.0 was released Jan 2014, so 10.0 will be supported for a year or so. Updating to 10.1 (when it is finalized in Q4) will very likely be painless, no incompatibilities, so as a practical matter, EOL is extended at least another year or so. This has been the history of FBSD since my exposure to 4.x back in 2000. A more recent example, FBSD 8.0 was announced Nov 2009. And now, nearly 5 years later, FBSD 8.4 is still a supported version. Sure, technically each minor version of 8.x reached EOL after a year. But since keeping an 8.x installation up-to-date has been so seamless, it achieves an EOL >=5 years from introduction. Anyway, how I see it based on real-world experience.

donavanm 12y ago

One aspect thats always stood at is the system/userland divide. The BSDs have both kernel and userland developed by the same folks. The leads to a consistent, enjoyable, user experience. In linux land the divide between kernel and user is quite stark. Throwing problems overthe fence, and blaming the "other side", is far too common. The state of documentation and man pages is indicative. Nearly everything in the BSDs has a well written up to date docs. Determining behavior in linux all too often comes down to reading opaque kernel code and old mailing list threads that may or may not be relevant.

_delirium 12y ago
That's the standard line, but I have never felt that as a user. Debian and FreeBSD feel about equally integrated to me as systems. Some stuff in both is developed by different upstreams (large parts of both the Debian and FreeBSD base installs are developed elsewhere), but the "buck stops here" when it comes to integration, integration-testing, engineering a consistent user experience, and generally making the system work for end-users. If anything Debian feels slightly more tightly integrated to me, though it depends on whether you frequently use a lot of non-base packages. FreeBSD's model is tight integration of the base install and light curation of everything else in userland, while Debian's model is to "Debianize" everything in the archive and ensure it plays well together, even non-base packages.

pjmlp 12y ago

The worse is that many now seem to equate GNU/Linux to UNIX, thus forgetting not all UNIX are alike, nor POSIX is a portable as it seems, X is not everywhere and many other differences.

clarry 12y ago
Another thing that bothers me a lot is that people equate GNU or Linux or GNU/Linux to free software.
_delirium 12y ago
I think market share has mostly led to that. As long as the commercial unixes had significant market share (esp. among paying customers), people developing Unix software had an incentive to write it portably, so it could run on at least Solaris and Linux. As the bottom dropped out of the Solaris market, a lot of packages decided not to bother, as >90% of Unix customers were on Linux. Some don't even bother supporting more than one specific Linux distribution (among commercial software packages, that's often RHEL, sometimes RHEL+Ubuntu). There was a period when non-x86 compatibility went through a similar low period (due to the decline of Alpha, SPARC, etc.), but increasing ARM market share is making architecture portability more practically relevant again.