6 ms·

Competition is good (not everyone using Linux I mean), and I've ran FreeBSD on my desktop and server for a few years. But whenever Netflix's use of FreeBSD com

Competition is good (not everyone using Linux I mean), and I've ran FreeBSD on my desktop and server for a few years. But whenever Netflix's use of FreeBSD comes up I never come away with a concrete understanding for: is the cost/performance Netflix gets with FreeBSD really not doable on Linux? I'm trying to understand if it's inertia, or if not, why more cloud companies with similar traffic (all the CDN companies for example) aren't also using FreeBSD somewhere in their stack. If it were just the case that they like FreeBSD that would be fine and I wouldn't ask more. But they mention in the slides FreeBSD is a performance-focused OS which seems to beg the question.

Thaxll 2y ago
The truth is that some senior engineer at Netflix chose to use FreeBSD and they stick to that idea since then, FreeBSD is not better it's happen to be the solution they chose. All the benefits they added to FreeBSD could be added the same way in Linux if it was missing. YouTube / Google CDN is much bigger than Netflix and runs 100% on Linux, you can make pretty much everything work on modern solution / languages / framework.
- inopinatus 2y ago
  Sorry, this is seriously misinformed: > YouTube / Google CDN is much bigger than Netflix Youtube and Netflix are on par. According to Sandvine, Netflix sneaked past Youtube in volume in 2023[1]. I believe their 2024 report shows them still neck-and-neck. > you can make pretty much everything work on modern solution Presenting a false equivalence without evidence is not convincing. "You could write it in SNOBOL and deploy it on TempleOS". Netflix didn't choose something arbitrary or by mistake. They chose one of the world's highest performing and most robustly tested infrastructure operating systems. It's the reason a FreeBSD derivative lies at the core of Juniper routers, Isilon and Netapp storage heads, every Playstation 3/4/5, and got mashed into NeXTSTEP to spawn Darwin and thence macOS, iOS etc. It continues to surprise me how folks in the tech sector routinely fail to notice how much BSD is deployed in the infrastructure around them. > All the benefits they added to FreeBSD could be added the same way in Linux They cannot. Case in point, the bisect analysis described in the presentation above doesn't exist for Linux, where userland distributions develop independently from the kernel. Netflix is touting here the value of FreeBSD's unified release, since the bisect process fundamentally relies on a single dimension of change (please ignore the mathematicians muttering about Schur's transform). [1] https://www.sandvine.com/press-releases/sandvines-2023-global-internet-phenomena-report-shows-24-jump-in-video-traffic-with-netflix-volume-overtaking-youtube https://www.sandvine.com/press-releases/sandvines-2023-globa...
  - zamadatix 2y ago
    > It's the reason a FreeBSD derivative lies at the core of Juniper routers Juniper is a particularly poor example to start the list with as they have been moving away from FreeBSD for years now with Junos OS Evolved being Linux based and specifically about how being based on FreeBSD was no longer a benefit to them.
    - pyvpx 2y ago
      no one seriously uses Evo in the way JunOS powers things. they’ve managed to package FreeBSD inside some Linux flavors. it’s wild, imo.
      - zamadatix 2y ago
        Guess I'm nobody, I think the PTX line is great ¯\_(ツ)_/¯.
  - davidw 2y ago
    > There's a reason it's the core of Juniper routers, Isilon and Netapp storage heads, every Playstation 3/4/5, and got mashed into NeXTSTEP to spawn macOS. Licensing? FreeBSD is a fine OS that surely has some advantages here and there, but I'm also inclined to think that big companies can make stuff work if they want to. PHP at Meta seems like a pretty good example of this.
    - papageek 2y ago
      FreeBSD or Linux management plane on routers/switches is mostly irrelevant. The majority of the data simply flows through the asics untouched.
    - inopinatus 2y ago
      The permissive, freewheeling nature of the BSD license is touted by some as an advantage for infrastructure use but in practice, Linux-based devices and services aren't particularly encumbered by the GPL, so to me it's a wash.
- secondcoming 2y ago
  'runs 100% on Linux' is a bit vague. What customisations do they do?
  - yencabulator 2y ago
    Google has ~9000 kernel patches, and a fairly custom userspace. Saving 2% at that scale is huge.
hiAndrewQuinn 2y ago
Might be a manpower thing. By hiring a bunch of FreeBSD Core devs, Netflix might be able to get a really talented team for cheaper than they might get a less ideologically flavored team. (I say this as I set up i3 on FreeBSD on my Thinkpad X280, I'm a big fan!)
- IshKebab 2y ago
  They also get much more control. If they employ most of the core FreeBSD devs they basically have their own OS that they can do what they like with. If they want some change that benefits their use case at the detriment of other people they can pretty much just do it. That's not really possible with Linux.
  - guenthert 2y ago
    The flip side is of course that in this scenario they would have to finance most improvements. In Linux land the cost of development is shared across many, which one might expect to yield a generally better product.
    - toast0 2y ago
      They're at the cutting edge. They're going to have to finance the development of most of this stuff anyway.
      - yencabulator 2y ago
        This development, but not all of the other things that go into a kernel and distro. If they pocket the whole core team, then they end up paying for all of the work, not just the few optimizations they really care about.
    - throw0101d 2y ago
      > In Linux land the cost of development is shared across many, which one might expect to yield a generally better product. And you have more cooks in the kitchen tweaking things that you also want to work on, so there's a higher chance of simply conflicts, but also folks that want to go in a completely different direction technically/philosophically.
toast0 2y ago
It's hard to do apples to apples comparison, because you'd need two excellent, committed teams working on this. I'm a FreeBSD supporter, but I'm sure you could get things to work in Linux too. I haven't seen any posts like 'yeah, we got 800 gbps out of our Linux CDN boxes too', but I don't see a lot of posts about CDN boxes at all. As someone else downthread wrote, using FreeBSD gives a lot of control, and IMHO provides a much more stable base to work with. Where I've worked, we didn't follow -CURRENT, and tended to avoid .0 releases, but it was rare to see breakage across upgrades and it was typically easy to track down what changed because there usually wasn't a lot of changes in the suspect system. That's not really been my experience with Linux. A small community probably helps get their changes upstreamed regularly too.
bluedino 2y ago
Are there papers out there from other companies that detail what performance levels have been achieved using Linux in a similar device to the Netflix OCA? Maybe they just use two devices that have 90% of the performance?
bbatha 2y ago
My understanding was that Netflix liked FreeBSD for a few reasons, some of them more historical than others. * The networking stack was faster at the time * dtrace * async sendfile(2) https://lists.freebsd.org/pipermail/svn-src-head/2016-January/080924.html https://lists.freebsd.org/pipermail/svn-src-head/2016-Januar... Could they have contributed async sendfile(2) to linux as well? Probably. In 2024 these advantages seem to be moot: ebpf, io_uring, more maturity in the linux network stack plus FreeBSD losing more and more vendor support by the day .
TheCondor 2y ago
The community is such that if one of either FreeBSD or Linux really outperformed the other in some capacity, they'd rally and get the gap closed pretty quickly. There are benchmarks that flatter both but they tend to converge and be pretty close. Netflix has a team of, I think it was 10 from the PDF, engineers working on this platform. A substantial number of them are FreeBSD contributors with relationships to it. It's a very special team. That's the difference maker here. If it was a team of former Red Hat guys, I'm sure they'd be on Linux. If it was a team of old Solaris guys, I wouldn't be surprised if they were on one of the Solaris offsprings. Then, Netflix knew that at their scale and to make it work, they had to build this stuff out. That was something they figured out pretty quickly, they found the right team and then built around them. It's far more sophisticated than "we replaced RedHat Enterprise with FreeBSD and reaped immediate performance advantages." At their scale, I don't think there is an off the shelf solution.
- kelsey98765431 2y ago
  I think you're close but missing an element. FreeBSD is a centrally designed and implemented OS from kernel to libc to userland. The entire OS is in the source tree, with updated and correct documentation, with coherent design. Any systems level team will prefer this form of tightly integrated solution to the systems layer problem if they are responsible for the highly built out specialized distributed application we call netflix. The reasons for design choices going all the way back to freebsd 1 are available on the mailing list in a central place. Everything is there. Trying to maintain your own linux distro is insanely difficult, infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years. The resources to match the relatively small freebsd design and implementation team are minuscule compared to the infinite crawl of linuxes, a team of 10 freebsd sysprogs is basically the same amount of people responsible for designing and building the entire thing. It comes down to the sources of truth. In the world of fbsd that's the single fbsd repo and the single mailing list. For a linux, that could be thousands of sources of truth across hundreds of platforms.
  - arp242 2y ago
    Practically this just doesn't matter all that much. You can prefer one approach to the other and that's all fine, but from a "serious engineering" perspective it just doesn't really matter. > Trying to maintain your own linux distro is insanely difficult, infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years. 2.2? Colour me skeptical on that. But it's really not that hard to make a Linux distro. I know, because I did it, and tons of other people have. It's little more than "get kernel + fuck about with Linux booting madness + bunch of userland stuff". The same applies to FreeBSD by the way, and according to the PDF "Netflix has an internal “distro”". The problems Google has is because they maintain extensive patchsets, not because they built their own distro. They would have the same problems with FreeBSD, or any other complex piece of software.

Thaxll 2y ago

The truth is that some senior engineer at Netflix chose to use FreeBSD and they stick to that idea since then, FreeBSD is not better it's happen to be the solution they chose. All the benefits they added to FreeBSD could be added the same way in Linux if it was missing. YouTube / Google CDN is much bigger than Netflix and runs 100% on Linux, you can make pretty much everything work on modern solution / languages / framework.

inopinatus 2y ago
Sorry, this is seriously misinformed: > YouTube / Google CDN is much bigger than Netflix Youtube and Netflix are on par. According to Sandvine, Netflix sneaked past Youtube in volume in 2023[1]. I believe their 2024 report shows them still neck-and-neck. > you can make pretty much everything work on modern solution Presenting a false equivalence without evidence is not convincing. "You could write it in SNOBOL and deploy it on TempleOS". Netflix didn't choose something arbitrary or by mistake. They chose one of the world's highest performing and most robustly tested infrastructure operating systems. It's the reason a FreeBSD derivative lies at the core of Juniper routers, Isilon and Netapp storage heads, every Playstation 3/4/5, and got mashed into NeXTSTEP to spawn Darwin and thence macOS, iOS etc. It continues to surprise me how folks in the tech sector routinely fail to notice how much BSD is deployed in the infrastructure around them. > All the benefits they added to FreeBSD could be added the same way in Linux They cannot. Case in point, the bisect analysis described in the presentation above doesn't exist for Linux, where userland distributions develop independently from the kernel. Netflix is touting here the value of FreeBSD's unified release, since the bisect process fundamentally relies on a single dimension of change (please ignore the mathematicians muttering about Schur's transform). [1] https://www.sandvine.com/press-releases/sandvines-2023-global-internet-phenomena-report-shows-24-jump-in-video-traffic-with-netflix-volume-overtaking-youtube https://www.sandvine.com/press-releases/sandvines-2023-globa...
- zamadatix 2y ago
  > It's the reason a FreeBSD derivative lies at the core of Juniper routers Juniper is a particularly poor example to start the list with as they have been moving away from FreeBSD for years now with Junos OS Evolved being Linux based and specifically about how being based on FreeBSD was no longer a benefit to them.
  - pyvpx 2y ago
    no one seriously uses Evo in the way JunOS powers things. they’ve managed to package FreeBSD inside some Linux flavors. it’s wild, imo.
    - zamadatix 2y ago
      Guess I'm nobody, I think the PTX line is great ¯\_(ツ)_/¯.
- davidw 2y ago
  > There's a reason it's the core of Juniper routers, Isilon and Netapp storage heads, every Playstation 3/4/5, and got mashed into NeXTSTEP to spawn macOS. Licensing? FreeBSD is a fine OS that surely has some advantages here and there, but I'm also inclined to think that big companies can make stuff work if they want to. PHP at Meta seems like a pretty good example of this.
  - papageek 2y ago
    FreeBSD or Linux management plane on routers/switches is mostly irrelevant. The majority of the data simply flows through the asics untouched.
  - inopinatus 2y ago
    The permissive, freewheeling nature of the BSD license is touted by some as an advantage for infrastructure use but in practice, Linux-based devices and services aren't particularly encumbered by the GPL, so to me it's a wash.
secondcoming 2y ago
'runs 100% on Linux' is a bit vague. What customisations do they do?
- yencabulator 2y ago
  Google has ~9000 kernel patches, and a fairly custom userspace. Saving 2% at that scale is huge.

hiAndrewQuinn 2y ago

Might be a manpower thing. By hiring a bunch of FreeBSD Core devs, Netflix might be able to get a really talented team for cheaper than they might get a less ideologically flavored team. (I say this as I set up i3 on FreeBSD on my Thinkpad X280, I'm a big fan!)

IshKebab 2y ago
They also get much more control. If they employ most of the core FreeBSD devs they basically have their own OS that they can do what they like with. If they want some change that benefits their use case at the detriment of other people they can pretty much just do it. That's not really possible with Linux.
- guenthert 2y ago
  The flip side is of course that in this scenario they would have to finance most improvements. In Linux land the cost of development is shared across many, which one might expect to yield a generally better product.
  - toast0 2y ago
    They're at the cutting edge. They're going to have to finance the development of most of this stuff anyway.
    - yencabulator 2y ago
      This development, but not all of the other things that go into a kernel and distro. If they pocket the whole core team, then they end up paying for all of the work, not just the few optimizations they really care about.
  - throw0101d 2y ago
    > In Linux land the cost of development is shared across many, which one might expect to yield a generally better product. And you have more cooks in the kitchen tweaking things that you also want to work on, so there's a higher chance of simply conflicts, but also folks that want to go in a completely different direction technically/philosophically.

toast0 2y ago

It's hard to do apples to apples comparison, because you'd need two excellent, committed teams working on this. I'm a FreeBSD supporter, but I'm sure you could get things to work in Linux too. I haven't seen any posts like 'yeah, we got 800 gbps out of our Linux CDN boxes too', but I don't see a lot of posts about CDN boxes at all. As someone else downthread wrote, using FreeBSD gives a lot of control, and IMHO provides a much more stable base to work with. Where I've worked, we didn't follow -CURRENT, and tended to avoid .0 releases, but it was rare to see breakage across upgrades and it was typically easy to track down what changed because there usually wasn't a lot of changes in the suspect system. That's not really been my experience with Linux. A small community probably helps get their changes upstreamed regularly too.

bluedino 2y ago

Are there papers out there from other companies that detail what performance levels have been achieved using Linux in a similar device to the Netflix OCA? Maybe they just use two devices that have 90% of the performance?

bbatha 2y ago

My understanding was that Netflix liked FreeBSD for a few reasons, some of them more historical than others. * The networking stack was faster at the time * dtrace * async sendfile(2) https://lists.freebsd.org/pipermail/svn-src-head/2016-January/080924.html https://lists.freebsd.org/pipermail/svn-src-head/2016-Januar... Could they have contributed async sendfile(2) to linux as well? Probably. In 2024 these advantages seem to be moot: ebpf, io_uring, more maturity in the linux network stack plus FreeBSD losing more and more vendor support by the day .

TheCondor 2y ago

The community is such that if one of either FreeBSD or Linux really outperformed the other in some capacity, they'd rally and get the gap closed pretty quickly. There are benchmarks that flatter both but they tend to converge and be pretty close. Netflix has a team of, I think it was 10 from the PDF, engineers working on this platform. A substantial number of them are FreeBSD contributors with relationships to it. It's a very special team. That's the difference maker here. If it was a team of former Red Hat guys, I'm sure they'd be on Linux. If it was a team of old Solaris guys, I wouldn't be surprised if they were on one of the Solaris offsprings. Then, Netflix knew that at their scale and to make it work, they had to build this stuff out. That was something they figured out pretty quickly, they found the right team and then built around them. It's far more sophisticated than "we replaced RedHat Enterprise with FreeBSD and reaped immediate performance advantages." At their scale, I don't think there is an off the shelf solution.

kelsey98765431 2y ago
I think you're close but missing an element. FreeBSD is a centrally designed and implemented OS from kernel to libc to userland. The entire OS is in the source tree, with updated and correct documentation, with coherent design. Any systems level team will prefer this form of tightly integrated solution to the systems layer problem if they are responsible for the highly built out specialized distributed application we call netflix. The reasons for design choices going all the way back to freebsd 1 are available on the mailing list in a central place. Everything is there. Trying to maintain your own linux distro is insanely difficult, infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years. The resources to match the relatively small freebsd design and implementation team are minuscule compared to the infinite crawl of linuxes, a team of 10 freebsd sysprogs is basically the same amount of people responsible for designing and building the entire thing. It comes down to the sources of truth. In the world of fbsd that's the single fbsd repo and the single mailing list. For a linux, that could be thousands of sources of truth across hundreds of platforms.
- arp242 2y ago
  Practically this just doesn't matter all that much. You can prefer one approach to the other and that's all fine, but from a "serious engineering" perspective it just doesn't really matter. > Trying to maintain your own linux distro is insanely difficult, infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years. 2.2? Colour me skeptical on that. But it's really not that hard to make a Linux distro. I know, because I did it, and tons of other people have. It's little more than "get kernel + fuck about with Linux booting madness + bunch of userland stuff". The same applies to FreeBSD by the way, and according to the PDF "Netflix has an internal “distro”". The problems Google has is because they maintain extensive patchsets, not because they built their own distro. They would have the same problems with FreeBSD, or any other complex piece of software.