3 ms·

I saw this deck on ZFS and btrfs while researching them some years ago [1]. It led me to think ZFS would be fine for me to run in my closet/personal cloud, but

by gen220 6y ago

I saw this deck on ZFS and btrfs while researching them some years ago [1]. It led me to think ZFS would be fine for me to run in my closet/personal cloud, but I’d probably want to avoid it at work. I know it’s an old deck... has anything substantial changed since it was put together? Specifically around the licensing [2], inclusion in the kernel, etc. It looks like FB in particular had invested a lot in btrfs [3] over ZFS. [1]: http://marc.merlins.org/linux/talks/Btrfs-LCA2015/Btrfs.pdf http://marc.merlins.org/linux/talks/Btrfs-LCA2015/Btrfs.pdf [2]: https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-on-mac-os-x-project-over-licensing-issues/ https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-o... [3]: https://btrfs.wiki.kernel.org/index.php/Contributors https://btrfs.wiki.kernel.org/index.php/Contributors

tw04 6y ago
Sure, I can help. That powerpoint is horribly out of date and I'd argue full of FUD (intentional or not). >• Raid 0, 1, 5, and 6 are also built in the filesystem BTRFS RAID has been a nightmare from day 1 including complete data loss. Parity-based RAID has been "just around the corner" for almost a DECADE. Sure you can do RAID-1 but in 2020 I'm just not interested in losing half of my capacity. Yes you can layer it on top of MDRAID but that eliminates half the elegance of what ZFS brought to the table. >ZFS is fairly memory hungry, it's recommended to have 16GB of RAM and give 8GB or more to ZFS (it wasn't designed to use the linux memory filesystem, so it uses its own memory that can't be shared with the rest of linux). This is just flat wrong. You need 2GB of memory for a happy filesystem, 8GB+ is if you're doing deduplication which is unnecessary overhead in most environments. I'm also not sure where the "memory that can't be shared with the rest of linux" is coming from - ARC will use and free memory as needed by the system. >Due to the CDDL being incompatible with GPLv2, a linux vendor or hardware vendor will never be able to ship a linux distribution or hardware device using ZFS Except Ubuntu already is. I believe SLES does as well. >As a result, you shouldn't plan on using ZFS for any product that you might ever want to ship one day. Delphix ships a product today based on ZFS. 0 issues. >Oracle may have stopped further work on ZFS as a result. Or it could be another reason entirely... Oracle absolutely didn't stop work on ZFS, I'm not even sure where he came up with that nonsense. Oracle continued to update and release new versions of ZFS long after the lawsuit was settled. You know what's far more telling? 13 YEARS after starting BTRFS: Oracle uses ext4 as their default filesystem, not BTRFS. Redhat has dropped support for BTRFS entirely. https://fossbytes.com/red-hat-deprecate-btrfs-filesystem-stratis/ https://fossbytes.com/red-hat-deprecate-btrfs-filesystem-str...
- gen220 6y ago
  Thank you very much for the play-by-play. This slide deck is from 2015, so I'm willing to give the author the benefit of the doubt, that things have probably changed a lot since then. It seems that the ZFS story has gotten much better since 2015 (FreeBSD & Ubuntu support, memory usage improvements, license problem proving to be not a big problem). Correspondingly, it seems that the btrfs story has regressed (redhat drops support, many issues in this greater thread raised around reliability and use cases, suspect development practices). My guess is that they were closer to parity in 2015. This deck was making the case for why btrfs would win out. The author seemingly turned out to be wrong, as we're all liable to be from time to time. My guess is that the licensing might be an issue for some big corporations today, but it appears to be generally benign, like you say. Thank you so much for the information! :)
  - zokula 6y ago
    ZFS sucks do not use it .
benlivengood 6y ago
ZFS is useful if you don't have any higher-level abstractions for bulk storage. Databases and object storage (including cloud storage) generally provide integrity, reliability, and snapshot features at least as well as ZFS. If you need robust posix/samba file servers then ZFS is a good choice. I've never worked on a site with a large fleet on ZFS; it always had its own niche. Maybe a good rule of thumb is that if you've ever needed to track down a missing file, fix file corruption, or have contingency plans for such an event in production then ZFS can help. If the default action is to wipe a host and reinstall if the local filesystem looks fishy then ZFS won't provide many benefits. Btrfs has the advantage of living in the kernel tree (but I use ZFS on Freebsd where the same is true), but still carries an aura of not being quite done. There are several "things seem to work well at version X, and watch out for Y" statements in https://btrfs.wiki.kernel.org/index.php/Incremental_Backup https://btrfs.wiki.kernel.org/index.php/Incremental_Backup for example. This is a year old at least so maybe all bugs and caveats are fixed. How could I be sure? No idea. zfs send and receive just work. Btrfs seems to be the filesystem to use if you can dedicate enough time to testing your specific use case thoroughly and keeping up to date with changes and improvements as opposed to ext4 or ZFS which change relatively infrequently. ZFS enables/tracks new features in the zpool metadata so there's a modicum of forward- and backward-compatibility. To be honest I am even a bit leery of ZFS-on-Linux but even Freebsd is moving there soon. Abandoning a working, trusted codebase is always scary.
secabeen 6y ago
It's the other way around. FB put a lot of investment into btrfs, but only for the features they needed. Single drive works great, because they needed that, but RAID-equivalents don't. On the flip-side, Sun->Oracle have spent decades investing in ZFS at all levels, and it's really solid now. Licensing is a minor issue with ZFS, but if you aren't distributing the code to others, those issues don't affect you.

by gen220 6y ago

tw04 6y ago

Sure, I can help. That powerpoint is horribly out of date and I'd argue full of FUD (intentional or not). >• Raid 0, 1, 5, and 6 are also built in the filesystem BTRFS RAID has been a nightmare from day 1 including complete data loss. Parity-based RAID has been "just around the corner" for almost a DECADE. Sure you can do RAID-1 but in 2020 I'm just not interested in losing half of my capacity. Yes you can layer it on top of MDRAID but that eliminates half the elegance of what ZFS brought to the table. >ZFS is fairly memory hungry, it's recommended to have 16GB of RAM and give 8GB or more to ZFS (it wasn't designed to use the linux memory filesystem, so it uses its own memory that can't be shared with the rest of linux). This is just flat wrong. You need 2GB of memory for a happy filesystem, 8GB+ is if you're doing deduplication which is unnecessary overhead in most environments. I'm also not sure where the "memory that can't be shared with the rest of linux" is coming from - ARC will use and free memory as needed by the system. >Due to the CDDL being incompatible with GPLv2, a linux vendor or hardware vendor will never be able to ship a linux distribution or hardware device using ZFS Except Ubuntu already is. I believe SLES does as well. >As a result, you shouldn't plan on using ZFS for any product that you might ever want to ship one day. Delphix ships a product today based on ZFS. 0 issues. >Oracle may have stopped further work on ZFS as a result. Or it could be another reason entirely... Oracle absolutely didn't stop work on ZFS, I'm not even sure where he came up with that nonsense. Oracle continued to update and release new versions of ZFS long after the lawsuit was settled. You know what's far more telling? 13 YEARS after starting BTRFS: Oracle uses ext4 as their default filesystem, not BTRFS. Redhat has dropped support for BTRFS entirely. https://fossbytes.com/red-hat-deprecate-btrfs-filesystem-stratis/ https://fossbytes.com/red-hat-deprecate-btrfs-filesystem-str...

gen220 6y ago
Thank you very much for the play-by-play. This slide deck is from 2015, so I'm willing to give the author the benefit of the doubt, that things have probably changed a lot since then. It seems that the ZFS story has gotten much better since 2015 (FreeBSD & Ubuntu support, memory usage improvements, license problem proving to be not a big problem). Correspondingly, it seems that the btrfs story has regressed (redhat drops support, many issues in this greater thread raised around reliability and use cases, suspect development practices). My guess is that they were closer to parity in 2015. This deck was making the case for why btrfs would win out. The author seemingly turned out to be wrong, as we're all liable to be from time to time. My guess is that the licensing might be an issue for some big corporations today, but it appears to be generally benign, like you say. Thank you so much for the information! :)
- zokula 6y ago
  ZFS sucks do not use it .

benlivengood 6y ago

ZFS is useful if you don't have any higher-level abstractions for bulk storage. Databases and object storage (including cloud storage) generally provide integrity, reliability, and snapshot features at least as well as ZFS. If you need robust posix/samba file servers then ZFS is a good choice. I've never worked on a site with a large fleet on ZFS; it always had its own niche. Maybe a good rule of thumb is that if you've ever needed to track down a missing file, fix file corruption, or have contingency plans for such an event in production then ZFS can help. If the default action is to wipe a host and reinstall if the local filesystem looks fishy then ZFS won't provide many benefits. Btrfs has the advantage of living in the kernel tree (but I use ZFS on Freebsd where the same is true), but still carries an aura of not being quite done. There are several "things seem to work well at version X, and watch out for Y" statements in https://btrfs.wiki.kernel.org/index.php/Incremental_Backup https://btrfs.wiki.kernel.org/index.php/Incremental_Backup for example. This is a year old at least so maybe all bugs and caveats are fixed. How could I be sure? No idea. zfs send and receive just work. Btrfs seems to be the filesystem to use if you can dedicate enough time to testing your specific use case thoroughly and keeping up to date with changes and improvements as opposed to ext4 or ZFS which change relatively infrequently. ZFS enables/tracks new features in the zpool metadata so there's a modicum of forward- and backward-compatibility. To be honest I am even a bit leery of ZFS-on-Linux but even Freebsd is moving there soon. Abandoning a working, trusted codebase is always scary.

secabeen 6y ago

It's the other way around. FB put a lot of investment into btrfs, but only for the features they needed. Single drive works great, because they needed that, but RAID-equivalents don't. On the flip-side, Sun->Oracle have spent decades investing in ZFS at all levels, and it's really solid now. Licensing is a minor issue with ZFS, but if you aren't distributing the code to others, those issues don't affect you.