6 ms·

Most small shops don't have the expertise -- or appetite to develop it -- to do an OS and kernel comparison in depth. Larger shops like Netflix do (and Facebook

Most small shops don't have the expertise -- or appetite to develop it -- to do an OS and kernel comparison in depth. Larger shops like Netflix do (and Facebook, etc). So if you want a detailed answer by a systems/kernel engineering team who have spent weeks studying it, you need to ask a larger shop. I'm one of those people. I'm on the Perf and OS team at Netflix. We're very familiar with FreeBSD -- we use it on our CDN. We're also very familiar with Linux -- we use it on our cloud. We're very familiar with a lot of other OSes too (Windows, AIX, Solaris, HP-UX, etc). So why don't we use BSD everywhere, or Linux everywhere? There's no single reason -- it's many reasons -- and it's serious engineering time to document them all and do the topic justice. I'd like to, but I don't have that time right now. But I wanted to point this out because I do see it commonly asked, and it sounds like a question that can be answered simply, but isn't. You're asking for engineering work, even to just summarize all the factors. I've written up an explanation of why not Solaris before on HN, since that's easy. But Linux vs FreeBSD is much more work, since they are both compelling options.

ComputerGuru 9y ago
No, you misunderstood me. I am not arguing for FreeBSD everywhere or Linux everywhere. To the contrary. I'm saying use whatever filesystem Linux provides when you need to use Linux. And use whatever operating system provides native support for ZFS when you need to use ZFS. We're using ZFS on FreeBSD to do our data storage, ASP.NET on Windows to run our business logic, and PHP under FreeBSD where PHP is required (previously HHVM on Linux because HHVM was the requirement). I'm saying pick the right OS for the right job. I'm saying figure out your priority with stability in mind. ZFS is a must when managing big data. It's nice to have otherwise. Decide which factor is the "must" in your case, and if it's ZFS, don't use Linux.
- tjoff 9y ago
  Problem is, Linux does not have any decent general purpose filesystem. That is why people risk their data using an untested hack.
  - anc84 9y ago
    What is not decent about ext4 as general purpose FS?
    - laumars 9y ago
      ext4 is fine for most people but it's not comparable to ZFS. Even ZFS aside, I've been unconvinced by ext4 these days and have starting using XFS on all new Linux server builds.
      - anc84 9y ago
        So it IS a decent general purpose FS. ;)
        laumars 9y ago
        I think there's two incompatible arguements going on here: 1. Is ext4 a decent general purpose file system? 2. Is ext4 a decent alternative or comparable to ZFS? My answers to those questions would be: 1. It's an "acceptable" general purpose file system but it's really starting to fall behind the competition. When I see comments like how people don't need features like snapshots et al I'm reminded about just how popular Windows Shadow Copy is. Ok that is a service rather than a file system feature but it shows that there is a demand for desktops to have this feature. Checksuming is another example - it's practically free service these days as CPUs have hardware extensions for popular checksum hashes. So the argument is akin to saying "why do we need file system journaling"; sure you don't need them but you're massively grateful when you do have them and your consumer hardware throws it's inevitable hissyfit. In my opinion Linux really does need to up its game. Ext4 feels almost stuck in time when compares to the likes of XFS which is years older than ext and yet they're working on snapshotting. Then you have really forward thinking file systems like what DragonflyBSD have been working on. They understand the need for better resilience of our data on desktops due to ever expanding storage capacities and HAMMERFS looks extremely exciting as a result. So to get back to your question. Yes ext4 is acceptable for most people, but is acceptable really good enough? Maybe we should invest more time and energy into XFS? 2. Is ext4 a decent alternative or comparable to ZFS? Simply put, no.
- brendangregg 9y ago
  We use zfsonlinux in production already. Have for years. It's had some issues, some have been fixed, some not yet. (I'm usually the go-to person to deal with the deepest ZFS issues.) Last I load tested FreeBSD ZFS it wasn't much different: a different set of issues. Some problems, some fixes. We're now using btrfs too. What's your technical beef with ZFS on Linux anyway?
  - laumars 9y ago
    He's not against ZoL. He's just saying that ZFS is a first class citizen on FreeBSD so if ZFS is amongst your concerns for that server, then picking an OS where ZFS if a first class citizen makes some logical sense. To be honest I do agree with him. I find FreeBSD to be close enough to Linux that any backend engineer or operator worth their salt should be able to swap between the two platforms. Nearly all the same tools that are available on Linux are available on FreeBSD, plus a few tools of it's own. I'm not advocating FreeBSD everywhere though - Linux has it's strengths too which would make it a better platform in other domains. But sometimes it feels like people turn to ZoL without even considering FreeBSD - which is a real pity as FreeBSD + ZFS is a dream to administrate. It's a lot easier than people expect and by not even entertaining the idea of FreeBSD I think they're missing an opportunity. This is all ideal world scenario stuff though. I do appreciate there are often other factors that complicate the decision making. Sometimes even political rather than technological factors.
    - sethhochberg 9y ago
      For us in particular, a big deciding factor was actually monitoring when we decided to consolidate our storage servers on ZFS on Linux (migrating a couple that were FreeBSD back to Linux). ZFS works well enough on either these days, and most of our linux folks are certainly comfortable enough in BSD-land, but our primary server monitoring stack at the time (NewRelic's server/infrastructure agents) doesn't have any BSD support at all and they didn't have interest in adding it, so we went back to the other set of tools we knew (linux) which integrated better with the other tooling we used and didn't have time to replace. The purists who try to suggest ZFS should be used from BSD whenever possible are probably forgetting "perfect is the enemy of good"
- wlll 9y ago
  I'm a bit confused by why ZFS shouldn't be used on Linux. It's just software and can be ported? In terms of using FreeBSD, I generally work in small (~6 person) sysadmin teams running hundreds of servers. There's already a lot of software and OSes to maintain and not a hell of a lot of time to do it, if we can cut out some of the complexity (making sure provisioning tools work on FreeBSD etc.) by just running as much stuff on Linux as possible then our jobs and lives become easier.
porker 9y ago
> So why don't we use BSD everywhere, or Linux everywhere? There's no single reason -- it's many reasons -- and it's serious engineering time to document them all and do the topic justice. I'd like to, but I don't have that time right now. If you ever have the time, that would be an awesome read. I find the differences between operating systems fascinating, and rarely done justice as people overlook the subtleties for a "Just use <<insert favourite tribe>>" approach.
- brendangregg 9y ago
  Part of the problem is that to do it justice, I need to get input from at least several other engineers and managers here. The choice of an OS is a big topic, and in the cloud it spans multiple teams (the Perf & OS team have the expertise and time to dig deep, but the choice also affects and needs input from the tools team, the platform security team, network engineering team, developers, etc.).
  - 4ad 9y ago
    I would be interested in such an in depth write up.
    - X86BSD 9y ago
      So would I, but I believe as he states it would be a monumental task across multiple disciplines and would that effort be worth it for probably a very small audience of nerds? :)
  - _-david-_ 9y ago
    Would it be possible to recommend it for a future Netflix tech blog?
olavgg 9y ago
I think you are spot on Brendan. We use both Linux and FreeBSD, and we see that applications can behave very differently in some scenarios on both platforms.

ComputerGuru 9y ago

No, you misunderstood me. I am not arguing for FreeBSD everywhere or Linux everywhere. To the contrary. I'm saying use whatever filesystem Linux provides when you need to use Linux. And use whatever operating system provides native support for ZFS when you need to use ZFS. We're using ZFS on FreeBSD to do our data storage, ASP.NET on Windows to run our business logic, and PHP under FreeBSD where PHP is required (previously HHVM on Linux because HHVM was the requirement). I'm saying pick the right OS for the right job. I'm saying figure out your priority with stability in mind. ZFS is a must when managing big data. It's nice to have otherwise. Decide which factor is the "must" in your case, and if it's ZFS, don't use Linux.

tjoff 9y ago
Problem is, Linux does not have any decent general purpose filesystem. That is why people risk their data using an untested hack.
- anc84 9y ago
  What is not decent about ext4 as general purpose FS?
  - laumars 9y ago
    ext4 is fine for most people but it's not comparable to ZFS. Even ZFS aside, I've been unconvinced by ext4 these days and have starting using XFS on all new Linux server builds.
    - anc84 9y ago
      So it IS a decent general purpose FS. ;)
      - laumars 9y ago
        I think there's two incompatible arguements going on here: 1. Is ext4 a decent general purpose file system? 2. Is ext4 a decent alternative or comparable to ZFS? My answers to those questions would be: 1. It's an "acceptable" general purpose file system but it's really starting to fall behind the competition. When I see comments like how people don't need features like snapshots et al I'm reminded about just how popular Windows Shadow Copy is. Ok that is a service rather than a file system feature but it shows that there is a demand for desktops to have this feature. Checksuming is another example - it's practically free service these days as CPUs have hardware extensions for popular checksum hashes. So the argument is akin to saying "why do we need file system journaling"; sure you don't need them but you're massively grateful when you do have them and your consumer hardware throws it's inevitable hissyfit. In my opinion Linux really does need to up its game. Ext4 feels almost stuck in time when compares to the likes of XFS which is years older than ext and yet they're working on snapshotting. Then you have really forward thinking file systems like what DragonflyBSD have been working on. They understand the need for better resilience of our data on desktops due to ever expanding storage capacities and HAMMERFS looks extremely exciting as a result. So to get back to your question. Yes ext4 is acceptable for most people, but is acceptable really good enough? Maybe we should invest more time and energy into XFS? 2. Is ext4 a decent alternative or comparable to ZFS? Simply put, no.
brendangregg 9y ago
We use zfsonlinux in production already. Have for years. It's had some issues, some have been fixed, some not yet. (I'm usually the go-to person to deal with the deepest ZFS issues.) Last I load tested FreeBSD ZFS it wasn't much different: a different set of issues. Some problems, some fixes. We're now using btrfs too. What's your technical beef with ZFS on Linux anyway?
- laumars 9y ago
  He's not against ZoL. He's just saying that ZFS is a first class citizen on FreeBSD so if ZFS is amongst your concerns for that server, then picking an OS where ZFS if a first class citizen makes some logical sense. To be honest I do agree with him. I find FreeBSD to be close enough to Linux that any backend engineer or operator worth their salt should be able to swap between the two platforms. Nearly all the same tools that are available on Linux are available on FreeBSD, plus a few tools of it's own. I'm not advocating FreeBSD everywhere though - Linux has it's strengths too which would make it a better platform in other domains. But sometimes it feels like people turn to ZoL without even considering FreeBSD - which is a real pity as FreeBSD + ZFS is a dream to administrate. It's a lot easier than people expect and by not even entertaining the idea of FreeBSD I think they're missing an opportunity. This is all ideal world scenario stuff though. I do appreciate there are often other factors that complicate the decision making. Sometimes even political rather than technological factors.
  - sethhochberg 9y ago
    For us in particular, a big deciding factor was actually monitoring when we decided to consolidate our storage servers on ZFS on Linux (migrating a couple that were FreeBSD back to Linux). ZFS works well enough on either these days, and most of our linux folks are certainly comfortable enough in BSD-land, but our primary server monitoring stack at the time (NewRelic's server/infrastructure agents) doesn't have any BSD support at all and they didn't have interest in adding it, so we went back to the other set of tools we knew (linux) which integrated better with the other tooling we used and didn't have time to replace. The purists who try to suggest ZFS should be used from BSD whenever possible are probably forgetting "perfect is the enemy of good"
wlll 9y ago
I'm a bit confused by why ZFS shouldn't be used on Linux. It's just software and can be ported? In terms of using FreeBSD, I generally work in small (~6 person) sysadmin teams running hundreds of servers. There's already a lot of software and OSes to maintain and not a hell of a lot of time to do it, if we can cut out some of the complexity (making sure provisioning tools work on FreeBSD etc.) by just running as much stuff on Linux as possible then our jobs and lives become easier.

porker 9y ago

> So why don't we use BSD everywhere, or Linux everywhere? There's no single reason -- it's many reasons -- and it's serious engineering time to document them all and do the topic justice. I'd like to, but I don't have that time right now. If you ever have the time, that would be an awesome read. I find the differences between operating systems fascinating, and rarely done justice as people overlook the subtleties for a "Just use <<insert favourite tribe>>" approach.

brendangregg 9y ago
Part of the problem is that to do it justice, I need to get input from at least several other engineers and managers here. The choice of an OS is a big topic, and in the cloud it spans multiple teams (the Perf & OS team have the expertise and time to dig deep, but the choice also affects and needs input from the tools team, the platform security team, network engineering team, developers, etc.).
- 4ad 9y ago
  I would be interested in such an in depth write up.
  - X86BSD 9y ago
    So would I, but I believe as he states it would be a monumental task across multiple disciplines and would that effort be worth it for probably a very small audience of nerds? :)
- _-david-_ 9y ago
  Would it be possible to recommend it for a future Netflix tech blog?

olavgg 9y ago

I think you are spot on Brendan. We use both Linux and FreeBSD, and we see that applications can behave very differently in some scenarios on both platforms.