7 ms·

The Future of ZFS on Ubuntu Desktop Is Not Looking Good

41 points by AdmiralAsshat 4y ago

theandrewbailey 4y ago
Is BTRFS rapidly gaining feature and reliability parity with ZFS?
- trillic 4y ago
  for desktop use-cases, I think so. I'm BTRFS everywhere at home now, even my NAS. All RAID1, backups to object storage.
- AdmiralAsshat 4y ago
  BTRFS seemingly still has problems, see: https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... but for desktop use, it's probably fine. It's been the default for new Fedora installations for over a year now, and I don't see too many Fedora users complaining about their filesystem being hosed. My current installation is still legacy EXT4 since I upgrade, but at some point I might blow it away and try a fresh install.
- adrian_b 4y ago
  BTRFS might be gaining feature and reliability parity with ZFS, but I have never heard of any advantage of BTRFS over ZFS and in all the benchmarks that I have ever seen BTRFS is much slower than ZFS. When the file system performance is important, XFS and EXT4 are the fastest. When the extra features provided by ZFS are important, it becomes the appropriate choice. For BTRFS, I am not aware of any application for which it would be the right choice.
  - quotemstr 4y ago
    Last time I checked, ZFS duplicated the OS page cache and didn't support defragmentation. Is that still the case?
    - vkaku 4y ago
      I understand the first one may be a genuine pptimization needed. But for the second one: when a file is striped across different physical disks with possibly different characteristics, how would you defrag it without knowing the read access pattern?
      - Dylan16807 4y ago
        > when a file is striped across different physical disks with possibly different characteristics, how would you defrag it without knowing the read access pattern? Looking at a single disk at a time, do your best to make sure all chunks of the same file are in order and adjacent. You could be smarter than that, but just that would do fine. The problem is that ZFS basically can't move things.
        ender341341 4y ago
        > The problem is that ZFS basically can't move things. The one killer feature for me with btrfs is offline deduping (and by extension not having de-duping require a super computer to run). My data has enough duplication that it'd be nice to have, but not worth requiring hundreds of gigs of ram. Being able to fire off a job after importing a large set of data (or maybe yearly-ish) would be awesome and give me a few extra years without having to update my hardware.
        vkaku 4y ago
        That doesn't account for physical disk characteristics at all. You could have disks from different generations all be part of a pool, CMR vs SMR(!) (needs zone resets and large chunk writes), disks could degrade etc. Plus, when you update a file, you literally create new links and update the file out of place (true for any COW FS). So the read won't be sequential if you want to access two versions at the same time. In that case you want to know if you optimize for hardware failure as opposed to for performance. This could be fixed for specific usecases but not all block patterns in general. Just why I posted this.
      - mappu 4y ago
  - nicolaslem 4y ago
    > I have never heard of any advantage of BTRFS over ZFS. The ability to add capacity to a pool by adding a single drive at a time, or the fact that it is readily available on any Linux box.
    - kadoban 4y ago
      > The ability to add capacity to a pool by adding a single drive at a time zfs supports that now
      - derkades 4y ago
        I think you are referring to RAIDZ expansion, which isn't merged yet? https://github.com/openzfs/zfs/pull/12225 https://github.com/openzfs/zfs/pull/12225
        deleted 4y ago
        [deleted]
        kadoban 4y ago
        Hm, thanks. Yeah, I thought it was already released. The thing I read said "implemented", I guess I took the wrong meaning from that.
  - intuxikated 4y ago
    Any application on linux where snapshots, resizable partitions, compression or deduplication is useful? Zfs on linux is still not ready for primetime on root partitions, and for most uses except raid5/6 (which isn't important for 99% of desktop users) btrfs is an excellent option There's a reason the other major non-debian distros have switched to it by default. I'm still waiting for ubuntu to allow installing using btrfs + fulldisk encryption by default, which is currently a massive issue.
    - aeadio 4y ago
      > Zfs on linux is still not ready for primetime on root partitions Not sure where you got this from. OpenZFS on Linux is considered stable by the project. It has been rock solid, free of major issues, and considered more than ready for prime time since at least 0.8, and probably much earlier. It’s now at 2.1. I’ve been using it across multiple environments, including my personal laptop and workstation which maintained the same root-on-ZFS install for 6+ years, including 2 full migrations to new hardware using zfs send. I’ve deployed much larger pools in professional settings and they’ve been rock solid. Btrfs has had numerous data loss issues in that time. I can only recall one data loss issue in a ZFS on Linux release, several years ago, which was caught very quickly.
      - intuxikated 4y ago
        Which major distro allows using ZFS from its official installer except ubuntu (now deprecated) and maybe arch linux? Which distro uses ZFS by default? I think almost none. Sure, ZFS might be stable for extra mass storage disks, but how many people use it as a filesystem for the linux root? Never seen anyone putting the main OS on zfs Btrfs hasn't had any major data loss issues for the last 5 years (excluding the raid issues) btrfs is much newer than ZFS, so obviously it took longer to be stable, if you're talking about 6+ years ago, Btrfs was obviously not ready yet, but now? It's been the default on OpenSUSE for 5 years and Fedora for 2.
        mustache_kimono 4y ago
        > Never seen anyone putting the main OS on zfs “I can’t believe Nixon won. I don’t know anyone who voted for him.” -- Pauline Kael I use ZFS on root. Know lots of people who do. Might be your bubble?
- orra 4y ago
  Depends on your use case, but for me, absolutely not. The btrfs manual says RAID5 “should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%.”
chunk_waffle 4y ago
The baked in ZFS support is what lead me to use Ubuntu on my Desktop for a few release cycles. Sure you can build the module yourself using most other distros but it was the path of least effort. Eventually, I realized that I don't really use ZFS' features much on my Desktop. It's nice for sure, but I just didn't find myself needing it as much as I do on my servers (where it's essential.) So I've left the Ubuntu camp anyway.
panick21_ 4y ago
This is really sad. I really hoped this would finally result in a fully integrated ZFS with a Linux distro. Maybe its just to much work? But then other distros have implemented many of the features with BTRFS so it can't be that much effort.
- pxc 4y ago
  > I really hoped this would finally result in a fully integrated ZFS with a Linux distro. What would this look like? In what ways do existing distros fail to live up to that vision?
  - vetinari 4y ago
    For me it means, that kernel, zfs, bootloader and initramfs-creating scripts are tested as a whole, by the distribution. For the user it means, that he can fearlessly update any of these packages and upon reboot, the system will successfully start. That is not given if you have one-off custom integration of these parts. For a while, I had a CentOS box with openzfs kABI modules; having to tinker after updates to mount the pool was not fun.
    - aeadio 4y ago
      Void Linux with ZFSBootMenu isn't far off. Multiple core Void team members uses and maintain ZFS support; kernel version bumps are blocked on OpenZFS support. ZFSBootMenu handles the bootloader, which can support fully upgraded ZFS pools and doesn't need pared-down zpool properties like Grub. ZFSBootMenu is developed natively on Void, and their wiki has bootstrap instructions for a Void install with root on ZFS. Everything's well supported, including native zpool encryption. Plus gives you a multitude of tools right in the bootloader -- snapshot rollback/management, boot environment creation/management, booting older kernels, and a full recovery command line with zfs available. You can tweak the bundled dracut config to include other tools as well. Even supports generating a unified, SecureBoot-signed EFI binary. The experience is leagues ahead of the Ubuntu mess.
      - 4y ago
1MachineElf 4y ago
Perhaps everyone who cared about Zsys/ZFS had already left Ubuntu due to other things, like Canonical's insistence on using the Snap (mis-)feature. Disappointing, because Zsys is quite good. I believe it would have gained prominence among competing ZFS boot environment managers if it hadn't been so tied to Ubuntu. Ubuntu was the first Linux distro that I could actually boot, back in 2007. Steam and ZFS were reasons for me to continue with it in 2020, but now, everything I want to do can be done better on NixOS.
deleted 4y ago
[deleted]
E39M5S62 4y ago
This is a good thing for ZFS and public perception of ZFS. Ubuntu has made some very poor choices with how they've integrated ZFS - * zsys has both implementation and design flaws. They utilize two zpools - one with limited feature flags that GRUB can use and one general pool, and then try to tightly couple datasets between those two pools while coordinating snapshots. zsys routinely runs a system out of space due to not cleaning up snapshots. It's been known to destroy user data as well. https://github.com/ubuntu/zsys/issues/196 https://github.com/ubuntu/zsys/issues/196 https://github.com/ubuntu/zsys/issues/218 https://github.com/ubuntu/zsys/issues/218 * Their packaging was and is still very sloppy. It's quite common to catch people running one version of kernel modules and another version of userland tools - often multiple releases apart. This has led to some serious problems. (no direct source for this, typically noted in libera.chat#zfsonlinux) * Ubuntu has created and applied bad patches to ZFS, resulting in destroyed pools. https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1906476 https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/190... If your exposure to ZFS is through Ubuntu and zsys, you'd naturally have a very poor opinion of it. ZFS, of course, isn't with out flaws. Ubuntu is not helping.
- aeadio 4y ago
  This right here. As a long time ZFS user, I was excited when Canonical promised first class ZFS integration. But it was clear from the first release that Ubuntu's ZFS project was a mess. Clearly some folks got excited about ZFS, but didn't absorb much of the existing knowledge or wisdom in the ecosystem. Issues like dataset/pool layout, how snapshots were handled on update, were handled in completely unusual and boneheaded ways. The ecosystem already had the concept of boot environments, which more or less standardized what a rool-on-zpool layout ought to look like, and how to handle graceful, recoverable system upgrades. What Ubuntu could have brought to the table was a really high quality beadm replacement for Linux, moving Grub's ZFS support forward with support for more feature flags and boot environments, or investing in ZFSBootMenu. Instead they went ahead with zsys, and it's still not entirely clear what they want or expect it to be. That they needed to write a daemon to enable "complex zfs dataset layouts" should've been an early clue that they were going down the wrong path. The dataset layout for a root filesystem should never be so complex, that you need to write a whole daemon to manage it. The packaging issues and bad patches are almost besides the point.
berry_sortoro 4y ago
[dead]

41 points by AdmiralAsshat 4y ago

theandrewbailey 4y ago

Is BTRFS rapidly gaining feature and reliability parity with ZFS?

trillic 4y ago
for desktop use-cases, I think so. I'm BTRFS everywhere at home now, even my NAS. All RAID1, backups to object storage.
AdmiralAsshat 4y ago
BTRFS seemingly still has problems, see: https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... but for desktop use, it's probably fine. It's been the default for new Fedora installations for over a year now, and I don't see too many Fedora users complaining about their filesystem being hosed. My current installation is still legacy EXT4 since I upgrade, but at some point I might blow it away and try a fresh install.
adrian_b 4y ago
BTRFS might be gaining feature and reliability parity with ZFS, but I have never heard of any advantage of BTRFS over ZFS and in all the benchmarks that I have ever seen BTRFS is much slower than ZFS. When the file system performance is important, XFS and EXT4 are the fastest. When the extra features provided by ZFS are important, it becomes the appropriate choice. For BTRFS, I am not aware of any application for which it would be the right choice.
- quotemstr 4y ago
  Last time I checked, ZFS duplicated the OS page cache and didn't support defragmentation. Is that still the case?
  - vkaku 4y ago
    I understand the first one may be a genuine pptimization needed. But for the second one: when a file is striped across different physical disks with possibly different characteristics, how would you defrag it without knowing the read access pattern?
    - Dylan16807 4y ago
      > when a file is striped across different physical disks with possibly different characteristics, how would you defrag it without knowing the read access pattern? Looking at a single disk at a time, do your best to make sure all chunks of the same file are in order and adjacent. You could be smarter than that, but just that would do fine. The problem is that ZFS basically can't move things.
      - ender341341 4y ago
        > The problem is that ZFS basically can't move things. The one killer feature for me with btrfs is offline deduping (and by extension not having de-duping require a super computer to run). My data has enough duplication that it'd be nice to have, but not worth requiring hundreds of gigs of ram. Being able to fire off a job after importing a large set of data (or maybe yearly-ish) would be awesome and give me a few extra years without having to update my hardware.
      - vkaku 4y ago
        That doesn't account for physical disk characteristics at all. You could have disks from different generations all be part of a pool, CMR vs SMR(!) (needs zone resets and large chunk writes), disks could degrade etc. Plus, when you update a file, you literally create new links and update the file out of place (true for any COW FS). So the read won't be sequential if you want to access two versions at the same time. In that case you want to know if you optimize for hardware failure as opposed to for performance. This could be fixed for specific usecases but not all block patterns in general. Just why I posted this.
    - mappu 4y ago
- nicolaslem 4y ago
  > I have never heard of any advantage of BTRFS over ZFS. The ability to add capacity to a pool by adding a single drive at a time, or the fact that it is readily available on any Linux box.
  - kadoban 4y ago
    > The ability to add capacity to a pool by adding a single drive at a time zfs supports that now
    - derkades 4y ago
      I think you are referring to RAIDZ expansion, which isn't merged yet? https://github.com/openzfs/zfs/pull/12225 https://github.com/openzfs/zfs/pull/12225
      - deleted 4y ago
        [deleted]
      - kadoban 4y ago
        Hm, thanks. Yeah, I thought it was already released. The thing I read said "implemented", I guess I took the wrong meaning from that.
- intuxikated 4y ago
  Any application on linux where snapshots, resizable partitions, compression or deduplication is useful? Zfs on linux is still not ready for primetime on root partitions, and for most uses except raid5/6 (which isn't important for 99% of desktop users) btrfs is an excellent option There's a reason the other major non-debian distros have switched to it by default. I'm still waiting for ubuntu to allow installing using btrfs + fulldisk encryption by default, which is currently a massive issue.
  - aeadio 4y ago
    > Zfs on linux is still not ready for primetime on root partitions Not sure where you got this from. OpenZFS on Linux is considered stable by the project. It has been rock solid, free of major issues, and considered more than ready for prime time since at least 0.8, and probably much earlier. It’s now at 2.1. I’ve been using it across multiple environments, including my personal laptop and workstation which maintained the same root-on-ZFS install for 6+ years, including 2 full migrations to new hardware using zfs send. I’ve deployed much larger pools in professional settings and they’ve been rock solid. Btrfs has had numerous data loss issues in that time. I can only recall one data loss issue in a ZFS on Linux release, several years ago, which was caught very quickly.
    - intuxikated 4y ago
      Which major distro allows using ZFS from its official installer except ubuntu (now deprecated) and maybe arch linux? Which distro uses ZFS by default? I think almost none. Sure, ZFS might be stable for extra mass storage disks, but how many people use it as a filesystem for the linux root? Never seen anyone putting the main OS on zfs Btrfs hasn't had any major data loss issues for the last 5 years (excluding the raid issues) btrfs is much newer than ZFS, so obviously it took longer to be stable, if you're talking about 6+ years ago, Btrfs was obviously not ready yet, but now? It's been the default on OpenSUSE for 5 years and Fedora for 2.
      - mustache_kimono 4y ago
        > Never seen anyone putting the main OS on zfs “I can’t believe Nixon won. I don’t know anyone who voted for him.” -- Pauline Kael I use ZFS on root. Know lots of people who do. Might be your bubble?
orra 4y ago
Depends on your use case, but for me, absolutely not. The btrfs manual says RAID5 “should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%.”

chunk_waffle 4y ago

The baked in ZFS support is what lead me to use Ubuntu on my Desktop for a few release cycles. Sure you can build the module yourself using most other distros but it was the path of least effort. Eventually, I realized that I don't really use ZFS' features much on my Desktop. It's nice for sure, but I just didn't find myself needing it as much as I do on my servers (where it's essential.) So I've left the Ubuntu camp anyway.

panick21_ 4y ago

This is really sad. I really hoped this would finally result in a fully integrated ZFS with a Linux distro. Maybe its just to much work? But then other distros have implemented many of the features with BTRFS so it can't be that much effort.

pxc 4y ago
> I really hoped this would finally result in a fully integrated ZFS with a Linux distro. What would this look like? In what ways do existing distros fail to live up to that vision?
- vetinari 4y ago
  For me it means, that kernel, zfs, bootloader and initramfs-creating scripts are tested as a whole, by the distribution. For the user it means, that he can fearlessly update any of these packages and upon reboot, the system will successfully start. That is not given if you have one-off custom integration of these parts. For a while, I had a CentOS box with openzfs kABI modules; having to tinker after updates to mount the pool was not fun.
  - aeadio 4y ago
    Void Linux with ZFSBootMenu isn't far off. Multiple core Void team members uses and maintain ZFS support; kernel version bumps are blocked on OpenZFS support. ZFSBootMenu handles the bootloader, which can support fully upgraded ZFS pools and doesn't need pared-down zpool properties like Grub. ZFSBootMenu is developed natively on Void, and their wiki has bootstrap instructions for a Void install with root on ZFS. Everything's well supported, including native zpool encryption. Plus gives you a multitude of tools right in the bootloader -- snapshot rollback/management, boot environment creation/management, booting older kernels, and a full recovery command line with zfs available. You can tweak the bundled dracut config to include other tools as well. Even supports generating a unified, SecureBoot-signed EFI binary. The experience is leagues ahead of the Ubuntu mess.
    - 4y ago

1MachineElf 4y ago

Perhaps everyone who cared about Zsys/ZFS had already left Ubuntu due to other things, like Canonical's insistence on using the Snap (mis-)feature. Disappointing, because Zsys is quite good. I believe it would have gained prominence among competing ZFS boot environment managers if it hadn't been so tied to Ubuntu. Ubuntu was the first Linux distro that I could actually boot, back in 2007. Steam and ZFS were reasons for me to continue with it in 2020, but now, everything I want to do can be done better on NixOS.

deleted 4y ago

[deleted]

E39M5S62 4y ago

This is a good thing for ZFS and public perception of ZFS. Ubuntu has made some very poor choices with how they've integrated ZFS - * zsys has both implementation and design flaws. They utilize two zpools - one with limited feature flags that GRUB can use and one general pool, and then try to tightly couple datasets between those two pools while coordinating snapshots. zsys routinely runs a system out of space due to not cleaning up snapshots. It's been known to destroy user data as well. https://github.com/ubuntu/zsys/issues/196 https://github.com/ubuntu/zsys/issues/196 https://github.com/ubuntu/zsys/issues/218 https://github.com/ubuntu/zsys/issues/218 * Their packaging was and is still very sloppy. It's quite common to catch people running one version of kernel modules and another version of userland tools - often multiple releases apart. This has led to some serious problems. (no direct source for this, typically noted in libera.chat#zfsonlinux) * Ubuntu has created and applied bad patches to ZFS, resulting in destroyed pools. https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1906476 https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/190... If your exposure to ZFS is through Ubuntu and zsys, you'd naturally have a very poor opinion of it. ZFS, of course, isn't with out flaws. Ubuntu is not helping.

aeadio 4y ago
This right here. As a long time ZFS user, I was excited when Canonical promised first class ZFS integration. But it was clear from the first release that Ubuntu's ZFS project was a mess. Clearly some folks got excited about ZFS, but didn't absorb much of the existing knowledge or wisdom in the ecosystem. Issues like dataset/pool layout, how snapshots were handled on update, were handled in completely unusual and boneheaded ways. The ecosystem already had the concept of boot environments, which more or less standardized what a rool-on-zpool layout ought to look like, and how to handle graceful, recoverable system upgrades. What Ubuntu could have brought to the table was a really high quality beadm replacement for Linux, moving Grub's ZFS support forward with support for more feature flags and boot environments, or investing in ZFSBootMenu. Instead they went ahead with zsys, and it's still not entirely clear what they want or expect it to be. That they needed to write a daemon to enable "complex zfs dataset layouts" should've been an early clue that they were going down the wrong path. The dataset layout for a root filesystem should never be so complex, that you need to write a whole daemon to manage it. The packaging issues and bad patches are almost besides the point.

berry_sortoro 4y ago

[dead]