9 ms·

> What also gets lost in the fray is FreeBSD, even with all those Linux-focused tools, is still a compelling and useful operating system Is FreeBSD compelling

by patrec 6y ago

> What also gets lost in the fray is FreeBSD, even with all those Linux-focused tools, is still a compelling and useful operating system Is FreeBSD compelling and useful at this point beyond as an inferior but more liberally licensed Linux substitute? This is basically what Sony (after their PS2 woes), Nintendo and Netflix (as far as I'm aware; maybe there are technical reasons too) have been using it for. The last clearly competent company I can remember picking FreeBSD for some technical benefits was WhatsApp many many years ago, and as far as I am aware that decision would not make sense today anymore.

trasz 6y ago
Netflix folks repeatedly stated that in their benchmarks Linux simply couldn’t push data fast enough.
- jaekash 6y ago
  Could be, could also be they did not understand how to configure Linux, would like to see something more concrete to back such a claim up. Personally I'm skeptical that it is the case.
  - gtirloni 6y ago
    > could also be they did not understand how to configure Linux They hired Brendan Gregg [0] so I think they know what's going on. 0 - http://www.brendangregg.com/ http://www.brendangregg.com/
    - jaekash 6y ago
      AFAIK Netflix has used FreeBSD since 2012[1] and Brendan Gregg only started working at FreeBSD in 2014[2]. So not sure what the point of your statement is. [1]: https://lists.freebsd.org/pipermail/freebsd-stable/2012-June/068129.html https://lists.freebsd.org/pipermail/freebsd-stable/2012-June... [2]: https://www.linkedin.com/in/brendangregg/ https://www.linkedin.com/in/brendangregg/
  - dijit 6y ago
    Even if that’s true then it remains a failing of the Linux ecosystem that they were unable to understand how to configure it correctly or that the defaults did not work as well as the FreeBSD ones. But I don’t think it’s the case as they’ve put a lot of work into hiring some of the brightest minds in OS development, including Brendan Gregg http://www.brendangregg.com/ http://www.brendangregg.com/
    - jaekash 6y ago
      > Even if that’s true then it remains a failing of the Linux ecosystem that they were unable to understand how to configure it correctly or that the defaults did not work as well as the FreeBSD ones. Or alternatively they just did not put the time in. No defaults cover all use cases, almost all software I have ever used in high load applications needed some tuning, this includes Solaris, MySQL and MongoDB among others. > But I don’t think it’s the case as they’ve put a lot of work into hiring some of the brightest minds in OS development, including Brendan Gregg AFAIK Netflix has used FreeBSD since 2012[1] and Brendan Gregg only started working at FreeBSD in 2014[2]. So not sure this really shows they knew what they were doing when they made the choice to go with FreeBSD. It could be they did but I still would like to something more than just Netflix's choice to back the claim up. Many more companies choose Linux. [1]: https://lists.freebsd.org/pipermail/freebsd-stable/2012-June/068129.html https://lists.freebsd.org/pipermail/freebsd-stable/2012-June... [2]: https://www.linkedin.com/in/brendangregg/ https://www.linkedin.com/in/brendangregg/
      - effie 6y ago
        Interesting, according to this presentation from 2013 there were several reasons to use BSD, but network speed is not among them. https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.pdf https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p...
        jaekash 6y ago
        This presentation also makes no indication that FreeBSD was chosen because it had better performance than Linux: https://youtu.be/KP_bKvXkoC4?t=732 https://youtu.be/KP_bKvXkoC4?t=732 Seems GPL already ruled Linux out.
  - aduitsis 6y ago
    Don't be. You are talking about the folks that pulled this off: https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf
    - jaekash 6y ago
      7 years or more after choosing FreeBSD. This in no way vouches for their expertise in 2012.
      - deleted 6y ago
        [deleted]
  - hylaride 6y ago
    I think that the engineers at Netflix "could figure it out". As a long time BSD fan (since 1998) who has had to accept the fact that Linux won out in the end, I can vouch that by far the biggest pain point on Linux is its networking stack. Whenever I have to do anything other than set an IP address I throw up in my mouth a little bit. Linux has/had its benefits (packaging, install base), but IPTables or its routing configuration isn't one of them. High performance is certainly achievable, but it has a lot of quirks. BSD's networking stack has always been more robust. Anybody who's worked heavily with both will tell you that. It did not surprise me one bit that the primary reason they chose FreeBSD over Linux for their CDN was networking. Juniper came to the same conclusion, on top of licensing reasons.
    - jaekash 6y ago
      > I think that the engineers at Netflix "could figure it out". And I would still like to see some evidence. > Whenever I have to do anything other than set an IP address I throw up in my mouth a little bit. Very compelling case you make there.
      - hylaride 6y ago
        > And I would still like to see some evidence. There are plenty of talks, papers, etc from both Juniper, Netflix, etc on why they made their choices, but your response is merely "could also be they did not understand how to configure Linux" without any evidence. We're not going to do your googling for you. > Very compelling case you make there. https://www.dictionary.com/browse/hyperbole https://www.dictionary.com/browse/hyperbole
        jaekash 6y ago
        > There are plenty of talks, papers, etc from both Juniper, Netflix, etc on why they made their choices, but your response is merely "could also be they did not understand how to configure Linux" without any evidence. Here are the only sources I could find that gave some reasons for Netflix's choice: - https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.pdf https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p... - https://youtu.be/KP_bKvXkoC4?t=732 https://youtu.be/KP_bKvXkoC4?t=732 Neither of them suggest FreeBSD was faster than Linux in their tests. From both of those sources I would say avoiding GPL was the primary consideration. If there is something to back your claim up, link to it. If you cannot link to it then how the heck do you expect me to find it. I'm not the person claiming it exists.
  - doublerabbit 6y ago
    The telecommunications company I am currently employed for uses FreeBSD for the backhaul applied with CG-NAT. And this is for the simple reason that the Linux stack couldn't keep up with the load.
  - Annatar 6y ago
    If FreeBSD performed better out of the box without needing as much or no tuning at all, then it's the better tool for the job, specifically their job.
    - jaekash 6y ago
      Maybe, still no evidence to back the speculation up.
      - Annatar 6y ago
        They have both Linux and FreeBSD, yet FreeBSD is the one they still use for streaming. A Linux-biased and centric shop like Netflix would ditch FreeBSD the first chance they had if it weren't better at it than Linux.
        jaekash 6y ago
        Both of these things suggest they had many other reasons to chose FreeBSD, and neither suggests FreeBSD was faster than Linux: - https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.pdf https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p... - https://youtu.be/KP_bKvXkoC4?t=732 https://youtu.be/KP_bKvXkoC4?t=732 Seems GPL was a problem for them. EDIT: another source suggesting GPL avoidance was the primary reason for choosing FreeBSD over Linux: https://youtu.be/vcyQBup-Gto?t=2731 https://youtu.be/vcyQBup-Gto?t=2731 - and the presenter also says that the performance comparisons they have done between Linux and FreeBSD is not exactly fair because they did not do all the engineering work on Linux that they did on FreeBSD.
- jaekash 6y ago
  https://youtu.be/vcyQBup-Gto?t=2739 https://youtu.be/vcyQBup-Gto?t=2739 Jonathan Looney, Engineering Manager at Netflix, claiming their primary reason for choosing FreeBSD was GPL avoidance and that subsequent performance testing (which was not the basis for their choice) is not exactly fair because they did not do the same amount of engineering work for Linux as they did for FreeBSD.
- zelly 6y ago
  Then they shouldn't be going through the kernel at all. FreeBSD will still be slower than accessing the NIC directly via DPDK.
dijit 6y ago
I used freebsd for my last project, I can’t deny that support for BSD is spotty on cloud providers; but as far as performance and so forth it’s not far behind if at all. The compelling argument for FreeBSD has always been: zfs, pf and dtrace. There’s also kqueues which are a much better interface to high throughput I/O than epoll. The problem is that every time this kind of question is asked the ecosystem in Linux is different. Right now Linux has bpftrace, ZFS (kinda) and IO_Uring; so the gap has closed considerably.
- lllr_finger 6y ago
  It's historically been a very similar proposition to Solaris: zfs, dtrace, containers, and higher performance. For as much as I like both FreeBSD and Solaris, Brendan Gregg (mentioned elsewhere in the comments) has a very convincing argument as to why the scales have tipped in favor of Linux in the last few years: http://www.brendangregg.com/blog/2017-09-05/solaris-to-linux-2017.html http://www.brendangregg.com/blog/2017-09-05/solaris-to-linux...
  - jaekash 6y ago
    ZFS integration was worse in Solaris 10 than in FreeBSD. In Solaris the page cache and ARC was not properly integrated and when using mmap data would be cached twice in both page cache and arc cache and there was weird issues with these contending with each other. I think this was still the case in Solaris 11 also but hard to say since it is closed source and their marketing material was fraudulent. As for the higher performance, I would like to see evidence that Solaris has better performance than Linux. The one saving grace of Solaris was that it gave you amazing debugging capabilities but the practical outcome of that was just giving you the ability to gain deep insight to see see exactly how bad Solaris was as an operating system. In all my years of using it (10+) I never experienced it as being more performant than Linux and in most cases it was way slower if you did not tune it significantly. There were some very good ideas that went into Solaris but they were almost invariably poorly implemented.
    - Annatar 6y ago

by patrec 6y ago

trasz 6y ago

Netflix folks repeatedly stated that in their benchmarks Linux simply couldn’t push data fast enough.

jaekash 6y ago
Could be, could also be they did not understand how to configure Linux, would like to see something more concrete to back such a claim up. Personally I'm skeptical that it is the case.
- gtirloni 6y ago
  > could also be they did not understand how to configure Linux They hired Brendan Gregg [0] so I think they know what's going on. 0 - http://www.brendangregg.com/ http://www.brendangregg.com/
  - jaekash 6y ago
    AFAIK Netflix has used FreeBSD since 2012[1] and Brendan Gregg only started working at FreeBSD in 2014[2]. So not sure what the point of your statement is. [1]: https://lists.freebsd.org/pipermail/freebsd-stable/2012-June/068129.html https://lists.freebsd.org/pipermail/freebsd-stable/2012-June... [2]: https://www.linkedin.com/in/brendangregg/ https://www.linkedin.com/in/brendangregg/
- dijit 6y ago
  Even if that’s true then it remains a failing of the Linux ecosystem that they were unable to understand how to configure it correctly or that the defaults did not work as well as the FreeBSD ones. But I don’t think it’s the case as they’ve put a lot of work into hiring some of the brightest minds in OS development, including Brendan Gregg http://www.brendangregg.com/ http://www.brendangregg.com/
  - jaekash 6y ago
    > Even if that’s true then it remains a failing of the Linux ecosystem that they were unable to understand how to configure it correctly or that the defaults did not work as well as the FreeBSD ones. Or alternatively they just did not put the time in. No defaults cover all use cases, almost all software I have ever used in high load applications needed some tuning, this includes Solaris, MySQL and MongoDB among others. > But I don’t think it’s the case as they’ve put a lot of work into hiring some of the brightest minds in OS development, including Brendan Gregg AFAIK Netflix has used FreeBSD since 2012[1] and Brendan Gregg only started working at FreeBSD in 2014[2]. So not sure this really shows they knew what they were doing when they made the choice to go with FreeBSD. It could be they did but I still would like to something more than just Netflix's choice to back the claim up. Many more companies choose Linux. [1]: https://lists.freebsd.org/pipermail/freebsd-stable/2012-June/068129.html https://lists.freebsd.org/pipermail/freebsd-stable/2012-June... [2]: https://www.linkedin.com/in/brendangregg/ https://www.linkedin.com/in/brendangregg/
    - effie 6y ago
      Interesting, according to this presentation from 2013 there were several reasons to use BSD, but network speed is not among them. https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.pdf https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p...
      - jaekash 6y ago
        This presentation also makes no indication that FreeBSD was chosen because it had better performance than Linux: https://youtu.be/KP_bKvXkoC4?t=732 https://youtu.be/KP_bKvXkoC4?t=732 Seems GPL already ruled Linux out.
- aduitsis 6y ago
  Don't be. You are talking about the folks that pulled this off: https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf
  - jaekash 6y ago
    7 years or more after choosing FreeBSD. This in no way vouches for their expertise in 2012.
    - deleted 6y ago
      [deleted]
- hylaride 6y ago
  I think that the engineers at Netflix "could figure it out". As a long time BSD fan (since 1998) who has had to accept the fact that Linux won out in the end, I can vouch that by far the biggest pain point on Linux is its networking stack. Whenever I have to do anything other than set an IP address I throw up in my mouth a little bit. Linux has/had its benefits (packaging, install base), but IPTables or its routing configuration isn't one of them. High performance is certainly achievable, but it has a lot of quirks. BSD's networking stack has always been more robust. Anybody who's worked heavily with both will tell you that. It did not surprise me one bit that the primary reason they chose FreeBSD over Linux for their CDN was networking. Juniper came to the same conclusion, on top of licensing reasons.
  - jaekash 6y ago
    > I think that the engineers at Netflix "could figure it out". And I would still like to see some evidence. > Whenever I have to do anything other than set an IP address I throw up in my mouth a little bit. Very compelling case you make there.
    - hylaride 6y ago
      > And I would still like to see some evidence. There are plenty of talks, papers, etc from both Juniper, Netflix, etc on why they made their choices, but your response is merely "could also be they did not understand how to configure Linux" without any evidence. We're not going to do your googling for you. > Very compelling case you make there. https://www.dictionary.com/browse/hyperbole https://www.dictionary.com/browse/hyperbole
      - jaekash 6y ago
        > There are plenty of talks, papers, etc from both Juniper, Netflix, etc on why they made their choices, but your response is merely "could also be they did not understand how to configure Linux" without any evidence. Here are the only sources I could find that gave some reasons for Netflix's choice: - https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.pdf https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p... - https://youtu.be/KP_bKvXkoC4?t=732 https://youtu.be/KP_bKvXkoC4?t=732 Neither of them suggest FreeBSD was faster than Linux in their tests. From both of those sources I would say avoiding GPL was the primary consideration. If there is something to back your claim up, link to it. If you cannot link to it then how the heck do you expect me to find it. I'm not the person claiming it exists.
- doublerabbit 6y ago
  The telecommunications company I am currently employed for uses FreeBSD for the backhaul applied with CG-NAT. And this is for the simple reason that the Linux stack couldn't keep up with the load.
- Annatar 6y ago
  If FreeBSD performed better out of the box without needing as much or no tuning at all, then it's the better tool for the job, specifically their job.
  - jaekash 6y ago
    Maybe, still no evidence to back the speculation up.
    - Annatar 6y ago
      They have both Linux and FreeBSD, yet FreeBSD is the one they still use for streaming. A Linux-biased and centric shop like Netflix would ditch FreeBSD the first chance they had if it weren't better at it than Linux.
      - jaekash 6y ago
        Both of these things suggest they had many other reasons to chose FreeBSD, and neither suggests FreeBSD was faster than Linux: - https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.pdf https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p... - https://youtu.be/KP_bKvXkoC4?t=732 https://youtu.be/KP_bKvXkoC4?t=732 Seems GPL was a problem for them. EDIT: another source suggesting GPL avoidance was the primary reason for choosing FreeBSD over Linux: https://youtu.be/vcyQBup-Gto?t=2731 https://youtu.be/vcyQBup-Gto?t=2731 - and the presenter also says that the performance comparisons they have done between Linux and FreeBSD is not exactly fair because they did not do all the engineering work on Linux that they did on FreeBSD.
jaekash 6y ago
https://youtu.be/vcyQBup-Gto?t=2739 https://youtu.be/vcyQBup-Gto?t=2739 Jonathan Looney, Engineering Manager at Netflix, claiming their primary reason for choosing FreeBSD was GPL avoidance and that subsequent performance testing (which was not the basis for their choice) is not exactly fair because they did not do the same amount of engineering work for Linux as they did for FreeBSD.
zelly 6y ago
Then they shouldn't be going through the kernel at all. FreeBSD will still be slower than accessing the NIC directly via DPDK.

dijit 6y ago

I used freebsd for my last project, I can’t deny that support for BSD is spotty on cloud providers; but as far as performance and so forth it’s not far behind if at all. The compelling argument for FreeBSD has always been: zfs, pf and dtrace. There’s also kqueues which are a much better interface to high throughput I/O than epoll. The problem is that every time this kind of question is asked the ecosystem in Linux is different. Right now Linux has bpftrace, ZFS (kinda) and IO_Uring; so the gap has closed considerably.

lllr_finger 6y ago
It's historically been a very similar proposition to Solaris: zfs, dtrace, containers, and higher performance. For as much as I like both FreeBSD and Solaris, Brendan Gregg (mentioned elsewhere in the comments) has a very convincing argument as to why the scales have tipped in favor of Linux in the last few years: http://www.brendangregg.com/blog/2017-09-05/solaris-to-linux-2017.html http://www.brendangregg.com/blog/2017-09-05/solaris-to-linux...
- jaekash 6y ago
  ZFS integration was worse in Solaris 10 than in FreeBSD. In Solaris the page cache and ARC was not properly integrated and when using mmap data would be cached twice in both page cache and arc cache and there was weird issues with these contending with each other. I think this was still the case in Solaris 11 also but hard to say since it is closed source and their marketing material was fraudulent. As for the higher performance, I would like to see evidence that Solaris has better performance than Linux. The one saving grace of Solaris was that it gave you amazing debugging capabilities but the practical outcome of that was just giving you the ability to gain deep insight to see see exactly how bad Solaris was as an operating system. In all my years of using it (10+) I never experienced it as being more performant than Linux and in most cases it was way slower if you did not tune it significantly. There were some very good ideas that went into Solaris but they were almost invariably poorly implemented.
  - Annatar 6y ago