6 ms·

Facebook uses Linux "where it is faster than FreeBSD, which is basically everywhere" per a recent whitepaper or presentation I saw. What I found interesting, is

by SEJeff 12y ago

Facebook uses Linux "where it is faster than FreeBSD, which is basically everywhere" per a recent whitepaper or presentation I saw. What I found interesting, is that Facebook runs FreeBSD in production at all. It just goes to show how solid the BSDs really are. I use the hell out of keepalived (OSS vrrp daemon) + conntrackd on Linux, but strongly prefer carpd and pf. It moves a lot slower, but it caters to a different audience.

krakensden 12y ago
whatsapp was a FreeBSD shop, and they weren't purchased that long ago.
- X-Istence 12y ago
  WhatsApp is still a FreeBSD shop AFAIK.
sturadnidge 12y ago
Be interested to see that paper / presentation if you can track it down, especially given https://www.facebook.com/careers/department?req=a0IA000000Cz53VMAR https://www.facebook.com/careers/department?req=a0IA000000Cz...
justincormack 12y ago
They have a job ad out [1] saying "Facebook is seeking a Linux Kernel Software Engineer to join our Kernel team, with a primary focus on the networking subsystem. Our goal over the next few years is for the Linux kernel network stack to rival or exceed that of FreeBSD." [1] https://www.facebook.com/careers/department?req=a0IA000000Cz53VMAR https://www.facebook.com/careers/department?req=a0IA000000Cz...
- contingencies 12y ago
  That's very interesting, but isn't very convincing. Is there some kind of study on the supposed differences? I like BSD, but I've always felt from admin experience that the Linux networking stack is far more capable in terms of features.
  - UNIXgod 12y ago
    BSD is the implementation to the networking stack.
    - contingencies 12y ago
      From the API design / pedigree side, sure. But that's a given, and a tangent to the curiosity I am trying to have sated :)
  - justincormack 12y ago
    Well obviously it is a brief statement. Linux generally does have enormous numbers of features, but they are no use if they do not apply to you. My guesses about Facebook is that network stack issues would be around ipv6, performance or large scale (eg lost of routes).
Keyframe 12y ago
Define faster though.
- SEJeff 12y ago
  There are 2 common things where "faster" makes sense. 1. Throughput 2. Latency Both are relevant for webapps like facebook.
allegory 12y ago
An anecdote. When the company I was working for at the time's AC failed in the machine room and melted three racks of kit, all 7 of our FreeBSD machines at the time were back online from scratch on hot spare kit in under two hours. Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place. Quality engineering, documentation and automation ability shits on everything else IMHO and that's where FreeBSD lands every time. Performance isn't a major thing for me. Linux on the other hand is heading in the direction of windows. I'll probably get flamed by the systemd proponents here but it took me three hours to work out how to debug this command: timedatectl set-ntp yes Which returned: Failed to issue message call
- incision 12y ago
  >'Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place.' I'd be overjoyed if I never had to admin Windows again, but it's a bit unfair to frame that as an OS comparison. The most basic of direct from TechNet 'best practices' would reduce the impact of a failure of that sort to minutes. That said, that sort of failure not at all uncommon. Further, I expect such disasters would happen far less often if an existing, poorly implemented Microsoft environment could be remediated into something better as directly as anything unix-like.
  - allegory 12y ago
    I think it's completely fair. Windows is completely non-deterministic due to the insane amount of coupling inside it and the sheer snowball of crap that has stuck to it over the years. In this case, they had tested DR strategy etc in place. Unfortunately when it came to doing it for real (onto the same backup hardware the DR plan was tested against), TSHTF and Windows threw an obscure COM error loading the AD catalog. That required contacting MS 1st line who didn't know what it was so had to escalate it to the AD guys. One obscure registry key change and an ACL change (related to ESE) and it was back. All best practices followed, yet complexity, poor design and secret knowledge crept in and shot the whole process. Stuff like that scares the shit out of me having been on the end of it way too many times now. Many a night have I spent up at 2AM trying to work out why the hell something odd has gone on inside Windows and taken something out in production. Not once in 20 years has a proper Unix (Solaris/FreeBSD) box woken me up or shafted me for hours.
jedberg 12y ago
reddit originally ran on FreeBSD. They switched to Linux before I got there for ease of administration. I wanted to switch it back to BSD, but never got around to it as it wasn't very high priority. I tried to switch again when we moved to Amazon, but sadly it wasn't an option so I went to Ubuntu instead. I've always thought that BSD is superior from an admin perspective, as in it's easier to do it right and keep it solid. I may be biased though as my first job was IT for a company that had BSD committers so everything we did was on BSD and I learned how to admin from arguably some of the best BSD admins in the world.
- lifeisstillgood 12y ago
  I would be interested in interviewing (for the worlds smallest blog audience) some of "the worlds best BSD admins" - is there any chance of an introduction ?
ams6110 12y ago
BSDs do change stuff too. OpenBSD recently replaced sendmail with OpenSMTPD, and apache with nginx. FreeBSD 10 has some big changes[1], depending on what you're doing with it. Still, in general I agree. I run OpenBSD as my main desktop system because it works and mostly stays out of my way. [1] http://www.freebsd.org/releases/10.0R/announce.html http://www.freebsd.org/releases/10.0R/announce.html
- Forbo 12y ago
  Just a heads-up, OpenBSD is actually removing nginx from the base install in favor of httpd. http://undeadly.org/cgi?action=article&sid=20140827065755 http://undeadly.org/cgi?action=article&sid=20140827065755 Edit: Adding in the link to the actual commit: http://marc.info/?l=openbsd-cvs&m=140908174910713&w=2 http://marc.info/?l=openbsd-cvs&m=140908174910713&w=2
  - ams6110 12y ago
    Wow that was pretty fast, by OpenBSD standards. nginx was just added to base a release or two ago. Thanks.
    - talideon 12y ago
      It's understandable when you know that httpd was spun out of relayd: they basically had almost all the work done and figured they'd might as well do the last few bits needed ot make it a standalone HTTP daemon.
- cjg_ 12y ago
  Actually, OpenBSD just replaced nginx with their own httpd.
  - Nux 12y ago
    What httpd would that be? I thought they replaced their custom Apache with (also custom) Nginx.

by SEJeff 12y ago

krakensden 12y ago

whatsapp was a FreeBSD shop, and they weren't purchased that long ago.

X-Istence 12y ago
WhatsApp is still a FreeBSD shop AFAIK.

sturadnidge 12y ago

Be interested to see that paper / presentation if you can track it down, especially given https://www.facebook.com/careers/department?req=a0IA000000Cz53VMAR https://www.facebook.com/careers/department?req=a0IA000000Cz...

justincormack 12y ago

They have a job ad out [1] saying "Facebook is seeking a Linux Kernel Software Engineer to join our Kernel team, with a primary focus on the networking subsystem. Our goal over the next few years is for the Linux kernel network stack to rival or exceed that of FreeBSD." [1] https://www.facebook.com/careers/department?req=a0IA000000Cz53VMAR https://www.facebook.com/careers/department?req=a0IA000000Cz...

contingencies 12y ago
That's very interesting, but isn't very convincing. Is there some kind of study on the supposed differences? I like BSD, but I've always felt from admin experience that the Linux networking stack is far more capable in terms of features.
- UNIXgod 12y ago
  BSD is the implementation to the networking stack.
  - contingencies 12y ago
    From the API design / pedigree side, sure. But that's a given, and a tangent to the curiosity I am trying to have sated :)
- justincormack 12y ago
  Well obviously it is a brief statement. Linux generally does have enormous numbers of features, but they are no use if they do not apply to you. My guesses about Facebook is that network stack issues would be around ipv6, performance or large scale (eg lost of routes).

Keyframe 12y ago

Define faster though.

SEJeff 12y ago
There are 2 common things where "faster" makes sense. 1. Throughput 2. Latency Both are relevant for webapps like facebook.

allegory 12y ago

An anecdote. When the company I was working for at the time's AC failed in the machine room and melted three racks of kit, all 7 of our FreeBSD machines at the time were back online from scratch on hot spare kit in under two hours. Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place. Quality engineering, documentation and automation ability shits on everything else IMHO and that's where FreeBSD lands every time. Performance isn't a major thing for me. Linux on the other hand is heading in the direction of windows. I'll probably get flamed by the systemd proponents here but it took me three hours to work out how to debug this command: timedatectl set-ntp yes Which returned: Failed to issue message call

incision 12y ago
>'Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place.' I'd be overjoyed if I never had to admin Windows again, but it's a bit unfair to frame that as an OS comparison. The most basic of direct from TechNet 'best practices' would reduce the impact of a failure of that sort to minutes. That said, that sort of failure not at all uncommon. Further, I expect such disasters would happen far less often if an existing, poorly implemented Microsoft environment could be remediated into something better as directly as anything unix-like.
- allegory 12y ago
  I think it's completely fair. Windows is completely non-deterministic due to the insane amount of coupling inside it and the sheer snowball of crap that has stuck to it over the years. In this case, they had tested DR strategy etc in place. Unfortunately when it came to doing it for real (onto the same backup hardware the DR plan was tested against), TSHTF and Windows threw an obscure COM error loading the AD catalog. That required contacting MS 1st line who didn't know what it was so had to escalate it to the AD guys. One obscure registry key change and an ACL change (related to ESE) and it was back. All best practices followed, yet complexity, poor design and secret knowledge crept in and shot the whole process. Stuff like that scares the shit out of me having been on the end of it way too many times now. Many a night have I spent up at 2AM trying to work out why the hell something odd has gone on inside Windows and taken something out in production. Not once in 20 years has a proper Unix (Solaris/FreeBSD) box woken me up or shafted me for hours.

jedberg 12y ago

reddit originally ran on FreeBSD. They switched to Linux before I got there for ease of administration. I wanted to switch it back to BSD, but never got around to it as it wasn't very high priority. I tried to switch again when we moved to Amazon, but sadly it wasn't an option so I went to Ubuntu instead. I've always thought that BSD is superior from an admin perspective, as in it's easier to do it right and keep it solid. I may be biased though as my first job was IT for a company that had BSD committers so everything we did was on BSD and I learned how to admin from arguably some of the best BSD admins in the world.

lifeisstillgood 12y ago
I would be interested in interviewing (for the worlds smallest blog audience) some of "the worlds best BSD admins" - is there any chance of an introduction ?

ams6110 12y ago

BSDs do change stuff too. OpenBSD recently replaced sendmail with OpenSMTPD, and apache with nginx. FreeBSD 10 has some big changes[1], depending on what you're doing with it. Still, in general I agree. I run OpenBSD as my main desktop system because it works and mostly stays out of my way. [1] http://www.freebsd.org/releases/10.0R/announce.html http://www.freebsd.org/releases/10.0R/announce.html

Forbo 12y ago
Just a heads-up, OpenBSD is actually removing nginx from the base install in favor of httpd. http://undeadly.org/cgi?action=article&sid=20140827065755 http://undeadly.org/cgi?action=article&sid=20140827065755 Edit: Adding in the link to the actual commit: http://marc.info/?l=openbsd-cvs&m=140908174910713&w=2 http://marc.info/?l=openbsd-cvs&m=140908174910713&w=2
- ams6110 12y ago
  Wow that was pretty fast, by OpenBSD standards. nginx was just added to base a release or two ago. Thanks.
  - talideon 12y ago
    It's understandable when you know that httpd was spun out of relayd: they basically had almost all the work done and figured they'd might as well do the last few bits needed ot make it a standalone HTTP daemon.
cjg_ 12y ago
Actually, OpenBSD just replaced nginx with their own httpd.
- Nux 12y ago
  What httpd would that be? I thought they replaced their custom Apache with (also custom) Nginx.