8 ms·

Whats the current status of the situation between the ZFS-on-Linux project and Linux 5.1+?

by kawsper 7y ago

burtonator 7y ago
How is ZFS in general vs btrfs? ZFS is basically killed by Oracle right?
- Wowfunhappy 7y ago
  > ZFS is basically killed by Oracle right? Uh, no? OpenZFS is being actively developed and maintained, albeit not by Oracle.
  - isatty 7y ago
    Being not by Oracle is a feature; not a bug.
- ComputerGuru 7y ago
  btrfs is a dumpster truck on fire and everyone serious about their data has switched to ZoL or is holding out in hopes that bcachefs won't be a similar clusterfsck (which shouldn't be too hard to do).
  - jolmg 7y ago
    Btrfs has such a bad rep right now, but because such comments like yours lack any specifics I don't know if I should worry if all my systems with a simple btrfs root partition are going to spontaneously corrupt one day. Can anyone give concrete examples of what's wrong with btrfs?
    - silversconfused 7y ago
      Lilo doesn't support it. Other than that I have no problem with it. I think it had some problems with being declared "prod ready" by some vendors with some certain kernel version, and then a bunch of redhat users went off and tried it with ancient kernels and lost data. To me, this is just ext4 all over again. That filesystem was _terrible_ in the early days. Ext4 is pretty ok now, just slow...
  - wahern 7y ago
    HAMMER2 is the outside contender. See https://en.wikipedia.org/wiki/HAMMER2 https://en.wikipedia.org/wiki/HAMMER2 (summary) and http://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEAD:/sys/vfs/hammer2/DESIGN http://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEAD... (design document) It's BSD licensed, among other useful characteristics.
- analogkid 7y ago
  No, OpenZFS is alive and thriving and we use it in production on many Centos systems. btrfs is a awful. We used it for a brief time on a production system with vanilla mirrored drives. Absolutely horrible. I will never use btrfs for anything ever again. I've been running unix systems for 30 years; even the darkest days of UFS was not that bad.
  - RussianCow 7y ago
    Could you elaborate a bit on what was awful about it?
    - analogkid 7y ago
      Sure. Our small ~500GB volume (on an OpenSuse system) became essentially read-only with ~50GB of free space on it. The error code was no free space. We clearly didn't have the "toolkit" needed to deal with such an issue. The box was a secured system, so we couldn't add an additional device easily (non-keyboard USB devices were disabled in the kernel, for example). Also, the read speed off the volume became terribly slow...it would've taken days to get the data off the device. We tried many commands to try to unbork the volume, with no luck. Even some of the btrfs commands were taking hours to run and became unkillable. We had to reboot the box several times... We had a 24 hr old copy of the data and we were able to lift the few very important modified files off the device. So we reformatted with XFS and were able to restore the volume to a working state. Fortunately, the failure happened late on a Friday so we had the weekend to put it all back together again. I have only hatred for btrfs. Complete crap. Only our business competitors should use it ;-) We only chose to use it on this small system just so that we could get meaningful experience with it. We also had it running on another system where the next weekend we took emergency downtime and removed btrfs from it as well. I've been using ZFS in production since it was released in Solaris 10 in 2006. It isn't perfect either. We did once hit a performance degradation which was fixed a few days later with a kernel patch. But that was 10 years ago. We never lost any data or sleep over it. We no longer have any Solaris boxes; they were all converted to Centos with ZFS. The main active ZFS pools are all 6TB of mirrored NVMe storage on systems with 2TB of ram and 48 core of INTC Platinum cpu's. Works like a charm.
- 2Ccltvcm 7y ago
  ZFS on Linux is where the majority of ZFS innovation is these days. Btrfs should be permanently deleted except for a sole copy in a museum somewhere as an example of a historical mistake negatively affecting millions of people.
  - eclipseo76 7y ago
    Why is everyone shitting on btrfs in this thread?
    - nisa 7y ago
      used it for an Hadoop cluster after I've read an interview with the author that it's ready for production and Suse talking that it's the premier filesystem for their enterprise distribution. At that time ZFS was at 0.6.x and the new ARC stuff was not there and no dkms for Ubuntu and I was afraid that nobody could handle that... A few month later I'm obsessed with reading any commit in master and the mailing list, because things like undeletable files, hard lockups of the whole cluster and manual reboots in rescue-image and running btrfsck and stuff like that started to took too much time. That beeing said no problems after 4.8 or so, besides out of space on 20gb used with lots of files in a 16tb raid-10 btrfs array but that was minor in comparision to the other problems... Might be unfair, but I've learned my lesson there. I decided to follow the spectactle from a large distance.
    - X-Istence 7y ago
      Those who have used it have many battle scars that lead to us never ever trusting it again.
    - Wowfunhappy 7y ago
      Apparently because quite a few people have had bad experiences with it? When filesystems go bad, they really go bad, so it makes sense that these users would feel motivated to share their experiences. Perhaps it's a vocal minority, but does it matter? Isn't the error rate what's most important? I've seen ZFS horror stories too, but not nearly as many, and that says something at least. Disclaimer: I've never used btrfs. I quite like ZFS.
- nisa 7y ago
  Broad general statements are always difficult but FWIW: - btrfs is fine if you want checksumming for your external HDD with some backup stuff that is mostly sequential i/o. - For everything else - especially anything that goes beyond a single disk use ZFS. btrfs RAID1 is not really raid1 but rather oddness of pid decides which drive to read from, raid5,6 had serious issues a while back - the write hole problem still exists for btrfs. ZFS stripe, mirror, raidz are well understood and work flawless afaik. btrfs also does strange things when replacing disks, like going read-only when one disk fails - at least it used to be that way. - performance is a mixed bag but broadly speaking for anything that is not sequential io on an empty disk btrfs is slower than other filesystems[1], for ZFS it depends can be slower or faster than ext4 depending on a lot of settings... - ZFS tooling is far superior to btrfs - I've used both a lot and you won't go back to btrfs tooling once you've used ZFS. - btrfs snapshots degrade performance with rising snapshot count. ZFS works fine. - ZFS has native encryption with zfs send / recv support in the upcoming 0.8.0. So you can do encrypted differential backups this way. - btrfs quota was for a long time broken and error-prone, not sure what the current status is but it never worked correctly for me. - ZFS lacks cp --reflink and linux pagecache integration as well as sendfile() on Linux and the ZFS ARC is independed of the pagecache - that has some consequences for e.g. mmap(). - ZVOLs are ZFS only you can create block devices as ZFS datasets i.e. zfs create -V 10G rpool/xfs && mkfs.xfs /dev/zvol/rpool/xfs && mount /mnt /dev/zvol/rpool/xfs can be useful for ceph, running docker on xfs volumes in container - btrfs works fine inside containers so you can run Docker in LXD/LXC - ZFS is not there yet - overlay2 support is also missing. - no io-scheduling via cgroups for ZFS afaik because the Linux queue subsystem is not used... - Last but not least: ZFS has ZIL & L2ARC. ZIL is huge for spinning disks because sync() writes are first written to an ssd storage and later async and sequential to the hdd. TL;DL: apt update and databases are fast even on spinning disks. L2ARC is an SSD-Cache but it's not so effective in my experience (low hitrate for typical hosting stuff) - Personally I will avoid using btrfs if I can. 1: https://www.phoronix.com/scan.php?page=article&item=freebsd-12-zfs&num=2 https://www.phoronix.com/scan.php?page=article&item=freebsd-... it's a little bit apples/oranges because ZFS Freebsd != Linux ZFS but IMHO it shows that the architecture/design is sound.
- zlynx 7y ago
  ZFS is OK I suppose, but it isn't native. So you have to jump through hoops to use it, just like Nvidia binary graphics drivers. BTRFS works well. I've used it for a NAS since 2012. It started with two mirrored 2 TB WD RED drives. Since then it's gone through an expansion to 6 drives in RAID 10 and two drive failures and replacements. Pretty decent performance and never a serious problem. I've also used it on two laptops with SSD. It had some weird edge cases there using Docker with a lot of snapshots and running out of space for metadata. Unlike some people, I found it pretty straight-forward to recover by adding more space using either a USB drive or reusing my swap partition. BTRFS snapshots are easy to do and great for backups and Docker image layers. I felt the need to write this because your comment got three or four anti-BTRFS responses, and BTRFS has always been good to me.
  - lsh 7y ago
    I agree, BTRFS has always been good to me as well and I use it day to day, however out of all the vague negativity in the comments below, this has real merit: > ... raid5,6 had serious issues a while back - the write hole problem still exists for btrfs. it does indeed. BTRFS should not be used on a RAID: https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
    - krageon 7y ago
      It shouldn't be used for RAID5 or 6. RAID0, 1 and 10 are fine.
nisa 7y ago
current git works fine - 5.1 compat fixes just landed - https://github.com/zfsonlinux/zfs/commits/master https://github.com/zfsonlinux/zfs/commits/master
josteink 7y ago
I don’t think the kernel people have reverted their ZoL-hostile changes, if that’s what you’re asking about. That said: ZoL works with Linux 5.0, but AFAIU that’s due to ZoL falling back to less optimal code. I know it works because I run both on my laptop.

Whats the current status of the situation between the ZFS-on-Linux project and Linux 5.1+?

burtonator 7y ago

How is ZFS in general vs btrfs? ZFS is basically killed by Oracle right?

Wowfunhappy 7y ago
> ZFS is basically killed by Oracle right? Uh, no? OpenZFS is being actively developed and maintained, albeit not by Oracle.
- isatty 7y ago
  Being not by Oracle is a feature; not a bug.
ComputerGuru 7y ago
btrfs is a dumpster truck on fire and everyone serious about their data has switched to ZoL or is holding out in hopes that bcachefs won't be a similar clusterfsck (which shouldn't be too hard to do).
- jolmg 7y ago
  Btrfs has such a bad rep right now, but because such comments like yours lack any specifics I don't know if I should worry if all my systems with a simple btrfs root partition are going to spontaneously corrupt one day. Can anyone give concrete examples of what's wrong with btrfs?
  - silversconfused 7y ago
    Lilo doesn't support it. Other than that I have no problem with it. I think it had some problems with being declared "prod ready" by some vendors with some certain kernel version, and then a bunch of redhat users went off and tried it with ancient kernels and lost data. To me, this is just ext4 all over again. That filesystem was _terrible_ in the early days. Ext4 is pretty ok now, just slow...
- wahern 7y ago
  HAMMER2 is the outside contender. See https://en.wikipedia.org/wiki/HAMMER2 https://en.wikipedia.org/wiki/HAMMER2 (summary) and http://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEAD:/sys/vfs/hammer2/DESIGN http://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEAD... (design document) It's BSD licensed, among other useful characteristics.
analogkid 7y ago
No, OpenZFS is alive and thriving and we use it in production on many Centos systems. btrfs is a awful. We used it for a brief time on a production system with vanilla mirrored drives. Absolutely horrible. I will never use btrfs for anything ever again. I've been running unix systems for 30 years; even the darkest days of UFS was not that bad.
- RussianCow 7y ago
  Could you elaborate a bit on what was awful about it?
  - analogkid 7y ago
    Sure. Our small ~500GB volume (on an OpenSuse system) became essentially read-only with ~50GB of free space on it. The error code was no free space. We clearly didn't have the "toolkit" needed to deal with such an issue. The box was a secured system, so we couldn't add an additional device easily (non-keyboard USB devices were disabled in the kernel, for example). Also, the read speed off the volume became terribly slow...it would've taken days to get the data off the device. We tried many commands to try to unbork the volume, with no luck. Even some of the btrfs commands were taking hours to run and became unkillable. We had to reboot the box several times... We had a 24 hr old copy of the data and we were able to lift the few very important modified files off the device. So we reformatted with XFS and were able to restore the volume to a working state. Fortunately, the failure happened late on a Friday so we had the weekend to put it all back together again. I have only hatred for btrfs. Complete crap. Only our business competitors should use it ;-) We only chose to use it on this small system just so that we could get meaningful experience with it. We also had it running on another system where the next weekend we took emergency downtime and removed btrfs from it as well. I've been using ZFS in production since it was released in Solaris 10 in 2006. It isn't perfect either. We did once hit a performance degradation which was fixed a few days later with a kernel patch. But that was 10 years ago. We never lost any data or sleep over it. We no longer have any Solaris boxes; they were all converted to Centos with ZFS. The main active ZFS pools are all 6TB of mirrored NVMe storage on systems with 2TB of ram and 48 core of INTC Platinum cpu's. Works like a charm.
2Ccltvcm 7y ago
ZFS on Linux is where the majority of ZFS innovation is these days. Btrfs should be permanently deleted except for a sole copy in a museum somewhere as an example of a historical mistake negatively affecting millions of people.
- eclipseo76 7y ago
  Why is everyone shitting on btrfs in this thread?
  - nisa 7y ago
    used it for an Hadoop cluster after I've read an interview with the author that it's ready for production and Suse talking that it's the premier filesystem for their enterprise distribution. At that time ZFS was at 0.6.x and the new ARC stuff was not there and no dkms for Ubuntu and I was afraid that nobody could handle that... A few month later I'm obsessed with reading any commit in master and the mailing list, because things like undeletable files, hard lockups of the whole cluster and manual reboots in rescue-image and running btrfsck and stuff like that started to took too much time. That beeing said no problems after 4.8 or so, besides out of space on 20gb used with lots of files in a 16tb raid-10 btrfs array but that was minor in comparision to the other problems... Might be unfair, but I've learned my lesson there. I decided to follow the spectactle from a large distance.
  - X-Istence 7y ago
    Those who have used it have many battle scars that lead to us never ever trusting it again.
  - Wowfunhappy 7y ago
    Apparently because quite a few people have had bad experiences with it? When filesystems go bad, they really go bad, so it makes sense that these users would feel motivated to share their experiences. Perhaps it's a vocal minority, but does it matter? Isn't the error rate what's most important? I've seen ZFS horror stories too, but not nearly as many, and that says something at least. Disclaimer: I've never used btrfs. I quite like ZFS.
nisa 7y ago
Broad general statements are always difficult but FWIW: - btrfs is fine if you want checksumming for your external HDD with some backup stuff that is mostly sequential i/o. - For everything else - especially anything that goes beyond a single disk use ZFS. btrfs RAID1 is not really raid1 but rather oddness of pid decides which drive to read from, raid5,6 had serious issues a while back - the write hole problem still exists for btrfs. ZFS stripe, mirror, raidz are well understood and work flawless afaik. btrfs also does strange things when replacing disks, like going read-only when one disk fails - at least it used to be that way. - performance is a mixed bag but broadly speaking for anything that is not sequential io on an empty disk btrfs is slower than other filesystems[1], for ZFS it depends can be slower or faster than ext4 depending on a lot of settings... - ZFS tooling is far superior to btrfs - I've used both a lot and you won't go back to btrfs tooling once you've used ZFS. - btrfs snapshots degrade performance with rising snapshot count. ZFS works fine. - ZFS has native encryption with zfs send / recv support in the upcoming 0.8.0. So you can do encrypted differential backups this way. - btrfs quota was for a long time broken and error-prone, not sure what the current status is but it never worked correctly for me. - ZFS lacks cp --reflink and linux pagecache integration as well as sendfile() on Linux and the ZFS ARC is independed of the pagecache - that has some consequences for e.g. mmap(). - ZVOLs are ZFS only you can create block devices as ZFS datasets i.e. zfs create -V 10G rpool/xfs && mkfs.xfs /dev/zvol/rpool/xfs && mount /mnt /dev/zvol/rpool/xfs can be useful for ceph, running docker on xfs volumes in container - btrfs works fine inside containers so you can run Docker in LXD/LXC - ZFS is not there yet - overlay2 support is also missing. - no io-scheduling via cgroups for ZFS afaik because the Linux queue subsystem is not used... - Last but not least: ZFS has ZIL & L2ARC. ZIL is huge for spinning disks because sync() writes are first written to an ssd storage and later async and sequential to the hdd. TL;DL: apt update and databases are fast even on spinning disks. L2ARC is an SSD-Cache but it's not so effective in my experience (low hitrate for typical hosting stuff) - Personally I will avoid using btrfs if I can. 1: https://www.phoronix.com/scan.php?page=article&item=freebsd-12-zfs&num=2 https://www.phoronix.com/scan.php?page=article&item=freebsd-... it's a little bit apples/oranges because ZFS Freebsd != Linux ZFS but IMHO it shows that the architecture/design is sound.
zlynx 7y ago
ZFS is OK I suppose, but it isn't native. So you have to jump through hoops to use it, just like Nvidia binary graphics drivers. BTRFS works well. I've used it for a NAS since 2012. It started with two mirrored 2 TB WD RED drives. Since then it's gone through an expansion to 6 drives in RAID 10 and two drive failures and replacements. Pretty decent performance and never a serious problem. I've also used it on two laptops with SSD. It had some weird edge cases there using Docker with a lot of snapshots and running out of space for metadata. Unlike some people, I found it pretty straight-forward to recover by adding more space using either a USB drive or reusing my swap partition. BTRFS snapshots are easy to do and great for backups and Docker image layers. I felt the need to write this because your comment got three or four anti-BTRFS responses, and BTRFS has always been good to me.
- lsh 7y ago
  I agree, BTRFS has always been good to me as well and I use it day to day, however out of all the vague negativity in the comments below, this has real merit: > ... raid5,6 had serious issues a while back - the write hole problem still exists for btrfs. it does indeed. BTRFS should not be used on a RAID: https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
  - krageon 7y ago
    It shouldn't be used for RAID5 or 6. RAID0, 1 and 10 are fine.

nisa 7y ago

current git works fine - 5.1 compat fixes just landed - https://github.com/zfsonlinux/zfs/commits/master https://github.com/zfsonlinux/zfs/commits/master

josteink 7y ago

I don’t think the kernel people have reverted their ZoL-hostile changes, if that’s what you’re asking about. That said: ZoL works with Linux 5.0, but AFAIU that’s due to ZoL falling back to less optimal code. I know it works because I run both on my laptop.