8 ms·

I went on a quest a few years ago, thinking it would be good for the industry to standardize on a single next generation filesystem for UNIX. I started with ZFS

I went on a quest a few years ago, thinking it would be good for the industry to standardize on a single next generation filesystem for UNIX. I started with ZFS on linux since that seemed to have the most vocal advocates. That lasted about a half year, until a bug in the code resulted in a completely corrupt disk, and I had to restore 4TB of data over a month from offside backups. That plus the licensing confusion around ZFS has made it impossible for ZFS to be the defacto choice. I went down the BTRFS path, despite it's dodgy reputation when netgear announced their little embedded NASes, and switched my server over to it. The experience was solid enough that I bought high-end synology and have had zero problems with it.

clSTophEjUdRanu 7y ago
I really don't understand the insane hype around ZFS. You can't read any thread that touches on filesystems without the ZFS zealots coming out.
- asveikau 7y ago
  I don't think I am a zealot, nor a heavy user, but I use it on 1 machine at home (an NFS server running FreeBSD, which I have clients for elsewhere in my house). I came to this idea when I saw some data loss on some magnetic disks in my house, and repairing or even assessing the level of damage was difficult. My experience is that it's pretty good. The tooling does what it says without a lot of drama. I can scrub while the system is in use and don't notice it mostly. I have seen some small corruptions that it was able to flag for me with specific filenames and fix. Snapshotting and send/receive is also very handy. I heard some people say they don't like to use it under heavy load. That seems reasonable to me. You're paying costs to get the integrity piece. So it's not for every use or every user. It is very good at what it does, however.
  - stiray 7y ago
    Same with me. I just figured out at some point, 10 years ago, that it is nice to have snapshots on root disk. And figured out FreeBSD is supporting ZFS. Tryed it, loved it, used it. The ZFS on linux was destabilized in latest versions (`ls /.zfs/snapshots`) and they blew it considerably by adding it to systemd (I need to reboot fedora multiple times before it boots ever since), but at least I know that my data are not lost (unlike btrfs, had two major crashes in two years). Quite frankly I'll rather wait for Raisser to get out of jail than use btrfs again. Anyway, I bet on Hammer2.
- yjftsjthsd-h 7y ago
  ZFS is mature/stable, its feature set is basically unmatched (data checksums, compression, atomic snapshots, RAID(0,1,10,5,6), send/receive) by any other option on Linux, and what competition it does have is unstable in some configurations (BTRFS), essentially dead in the water (reiserfs), in early development (bcachefs), or far more complex to manage (gluster, ceph, LVM+XFS). Other than the licensing issue, ZFS is basically a silver bullet.
  - bbatha 7y ago
    Also don't underrate good documentation and easy to use tooling.
    - somehnguy 7y ago
      This was the killer feature for me. I had been wanting to try ZFS on my home NAS for a while (for snapshotting/redundancy/data integrity) and finally got enough disks that it made sense. I wasn't looking forward to learning what I presumed to be a very complicated system though. About 15 minutes into my research for setting up and maintaining a ZFS filesystem and I just went - wait thats it? So incredibly simple and well documented, it has been a joy to use. It is very rarely that complicated operations on complicated systems use such simple and easy to understand commands. It just does what I expect!
      - danudey 7y ago
        ZFS is incredibly easy to learn to use, whereas btrfs is quite complicated to learn/use, and even more so if you've used ZFS since a lot of things are either just different enough to be weird, or so different that it makes no sense. Examples: ZFS snapshots can be recursive (-r) or not, whereas on btrfs they cannot be recursive; in discussions I've seen, this is mentioned as "a feature", since you can create a subvolume for data that you don't want to be part of the snapshot, but it also prevents you from dividing up a logical heirarchy into multiple behaviours (compression vs. not, block size, etc.).
        7y ago
- tjoff 7y ago
  The hype is quite easy to understand. Snapshots and checksums are two complete game-changers. ZFS has them both. And there are no real alternatives in many cases. I've personally waited for BTRFS longer than a decade but my use-cases are yet to be considered stable (not something you really mess with in regard to filesystems). Honestly, as sure as I have been on the success of BTRFS I now consider BTRFS dead on arrival - if it will ever even arrive. The pace of development is slower than the universe around it, that might be too harsh but really - no RAID6 yet? A decade ago the impression I got was "soon". And now 2-drive parity is becoming obsolete. ZFS has tons of warts for home-use, I agree. So, for a home-user with high demands I don't see anything exciting in the future.
  - fulafel 7y ago
    lvm and hence ext4 etc have had snapshots for ages.
    - tjoff 7y ago
      As do NTFS. But they are not really comparable to "real" filesystem snapshotting, at least not in my opinion.
  - TurningCanadian 7y ago
    There were a bunch of btrfs raid56 patches last year. I think the known bugs have been addressed and is just that the wiki page hasn't been updated. Re obsolete, are you referring to RAID1C3?
    - tjoff 7y ago
      I'm thinking of this: https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/ https://www.zdnet.com/article/why-raid-5-stops-working-in-20... I'd much prefer something like raidz3 compared to the authors setup. RAID1C3 is nice but very expensive for use in bulk storage at home.
  - mycall 7y ago
    What warts do you speak of?
- somehnguy 7y ago
  Have you tried it? I went from having never used ZFS to loving it (and I guess being one of those zealots) very quickly after setting it up. So simple yet so powerful!
- kqr 7y ago
  ZFS is the worst filesystem/volume manager, except all others.
- twic 7y ago
  Eh, i'm waiting for them to rewrite it in Rust.
  - paulddraper 7y ago
    Poe's law
  - doctor_eval 7y ago
    This is such a meta-comment that I actually LOLd!
- rnd0 7y ago
  >You can't read any thread that touches on filesystems without the ZFS zealots coming out. Agreed 100%. That's particularly annoying to us desktop users. It took me years to figure out that no, FreeBSD aside, it doesn't bring anything to the table outside of enterprise storage use cases. At least it doesn't bring anything that's worth the hassles (I don't have to export ntfs filesystems before using them on another computer; same for ext4 -and then there's performance).
- seized 7y ago
  ZFS is like really good snow tires in the winter. You can tell people with other tires how great it is to have really good tires, but they dont believe you until they experience the benefits for themselves. Or put another way, no one "needs" ZFS until they really need it, then they wont live without it ever again. I switched to it after the 7200.11 firmware mess, where the drives reported successful writes but didnt write anything. ZFS would have caught that, my Adaptec card certainly couldnt have and didnt. ZFS to the rescue again a while later when those (now firmware updated) 7200.11 drives started dropping after 15k hours of service. ZFS saved my data when two drives started failing in my RAID5 set at the same time. All the weird minor problems that would cause random issues or performance issues for other file systems like flaky SATA cables, intermittent HBA/backplane ports, etc. ZFS catches them all and informs you. Having been hit by bit rot, corrupted files, corrupted file systems, etc etc before switching, ZFS is fantastic. And there is something great about watching it scrub at >1GB/sec, verifying every single bit of your data.
- ksec 7y ago
  I had to check the dictionary for the meaning of "hype" a situation in which something is advertised and discussed in newspapers, on television, etc. a lot in order to attract everyone's interest: May be its just me because Morden day usage of "hype" seems to involve and implies a negative meaning, especially in tech. Similar to false advertising. And no one was actively promoting ZFS, they were only very "responsive". And then zealots, I had to reread 226 comments, ran to Cambridge dictionary a person who has very strong opinions about something, and tries to make other people have them too I dont see anyone having strong opinions and force others to have the same. If anything a lot of people are showing not because the love ZFS, but they have been burnt by btrfs.
zionic 7y ago
I have a synology NAS on btrfs. One of the best computer purchases I've ever made.
- lostlogin 7y ago
  I’ll second this, it’s fantastic. The time it takes to expand when adding a second 16TB is deeply average (8 days) but that’s about it for downsides. It’s the best computer I’ve owned.
m4rtink 7y ago
Hard to standardize on something that can't be maintained in the same place all your other filesystems are in (in the Kernel) for licensing reasons.
- rbanffy 7y ago
  Only the boot file system drivers need to be in the kernel. As long as there is a stable ABI, it's fine for everything else to be someplace else.
  - skissane 7y ago
    > As long as there is a stable ABI, it's fine for everything else to be someplace else. Mainline Linux has a policy against in-kernel ABI stability guarantees. User-space is given ABI stability guarantees, in-kernel code by intention is not. That includes filesystems.
starfallg 7y ago
Btrfs is the only FS I used that resulted in complete FS corruption losing nearly all data on disk, not once, but 3 times. After that, none of the features like compression, snapshots, COW or checksums meant anything to me. I'm much happier with ext4 and xfs on lvm.
- RX14 7y ago
  It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. One possible cause is distro choice tends to be per person and how up to date said distro keeps it's kernel. But, I'm not sure.
  - thfuran 7y ago
    I think the probable cause is that it's not common bugs that cause the corruption but uncommon ones. Most of the time, they work fine. But you really want a stronger guarantee than that out of your filesystem.
    - SEJeff 7y ago
      Historically, the biggest bugs in btrfs were when you came close to filling up the filesystem. For the longest time, you'd get -ENOSPC (no space left) even when you had many Gb of space left due to really bad metadata and block level space usage.
      - blattimwind 7y ago
        Let's not forget about various performance issues which were exacerbated by "low free space" conditions (i.e. after you filled the volume beyond 80 % these started to pop up). A file system that will sometimes go down well into the fractional IOPS range is not very useful. Some of these are fixed by now, though.
tw04 7y ago
Legitimately curious what the ZFS bug was. I’ve not heard of a TFDL bug in zfs for a Loooong time. The reason Synology btrfs is mostly solid is because they refused to ever use the btrfs raid layer. But the second you move to btrfs on LVM you lose a large portion of the supposed benefits. Having used both, never lost data on zfs and I’ve been using it since it was released and have had it save me from silent data corruption. BTRFS hasn’t ever lost me an entire file system, but I’ve definitely lost files.
pizza234 7y ago
Do you have a link to the bug issue? ZFS purpotedly never had any corruption issues on release versions, so that makes it a really interesting case.

clSTophEjUdRanu 7y ago

I really don't understand the insane hype around ZFS. You can't read any thread that touches on filesystems without the ZFS zealots coming out.

asveikau 7y ago
I don't think I am a zealot, nor a heavy user, but I use it on 1 machine at home (an NFS server running FreeBSD, which I have clients for elsewhere in my house). I came to this idea when I saw some data loss on some magnetic disks in my house, and repairing or even assessing the level of damage was difficult. My experience is that it's pretty good. The tooling does what it says without a lot of drama. I can scrub while the system is in use and don't notice it mostly. I have seen some small corruptions that it was able to flag for me with specific filenames and fix. Snapshotting and send/receive is also very handy. I heard some people say they don't like to use it under heavy load. That seems reasonable to me. You're paying costs to get the integrity piece. So it's not for every use or every user. It is very good at what it does, however.
- stiray 7y ago
  Same with me. I just figured out at some point, 10 years ago, that it is nice to have snapshots on root disk. And figured out FreeBSD is supporting ZFS. Tryed it, loved it, used it. The ZFS on linux was destabilized in latest versions (`ls /.zfs/snapshots`) and they blew it considerably by adding it to systemd (I need to reboot fedora multiple times before it boots ever since), but at least I know that my data are not lost (unlike btrfs, had two major crashes in two years). Quite frankly I'll rather wait for Raisser to get out of jail than use btrfs again. Anyway, I bet on Hammer2.
yjftsjthsd-h 7y ago
ZFS is mature/stable, its feature set is basically unmatched (data checksums, compression, atomic snapshots, RAID(0,1,10,5,6), send/receive) by any other option on Linux, and what competition it does have is unstable in some configurations (BTRFS), essentially dead in the water (reiserfs), in early development (bcachefs), or far more complex to manage (gluster, ceph, LVM+XFS). Other than the licensing issue, ZFS is basically a silver bullet.
- bbatha 7y ago
  Also don't underrate good documentation and easy to use tooling.
  - somehnguy 7y ago
    This was the killer feature for me. I had been wanting to try ZFS on my home NAS for a while (for snapshotting/redundancy/data integrity) and finally got enough disks that it made sense. I wasn't looking forward to learning what I presumed to be a very complicated system though. About 15 minutes into my research for setting up and maintaining a ZFS filesystem and I just went - wait thats it? So incredibly simple and well documented, it has been a joy to use. It is very rarely that complicated operations on complicated systems use such simple and easy to understand commands. It just does what I expect!
    - danudey 7y ago
      ZFS is incredibly easy to learn to use, whereas btrfs is quite complicated to learn/use, and even more so if you've used ZFS since a lot of things are either just different enough to be weird, or so different that it makes no sense. Examples: ZFS snapshots can be recursive (-r) or not, whereas on btrfs they cannot be recursive; in discussions I've seen, this is mentioned as "a feature", since you can create a subvolume for data that you don't want to be part of the snapshot, but it also prevents you from dividing up a logical heirarchy into multiple behaviours (compression vs. not, block size, etc.).
      - 7y ago
tjoff 7y ago
The hype is quite easy to understand. Snapshots and checksums are two complete game-changers. ZFS has them both. And there are no real alternatives in many cases. I've personally waited for BTRFS longer than a decade but my use-cases are yet to be considered stable (not something you really mess with in regard to filesystems). Honestly, as sure as I have been on the success of BTRFS I now consider BTRFS dead on arrival - if it will ever even arrive. The pace of development is slower than the universe around it, that might be too harsh but really - no RAID6 yet? A decade ago the impression I got was "soon". And now 2-drive parity is becoming obsolete. ZFS has tons of warts for home-use, I agree. So, for a home-user with high demands I don't see anything exciting in the future.
- fulafel 7y ago
  lvm and hence ext4 etc have had snapshots for ages.
  - tjoff 7y ago
    As do NTFS. But they are not really comparable to "real" filesystem snapshotting, at least not in my opinion.
- TurningCanadian 7y ago
  There were a bunch of btrfs raid56 patches last year. I think the known bugs have been addressed and is just that the wiki page hasn't been updated. Re obsolete, are you referring to RAID1C3?
  - tjoff 7y ago
    I'm thinking of this: https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/ https://www.zdnet.com/article/why-raid-5-stops-working-in-20... I'd much prefer something like raidz3 compared to the authors setup. RAID1C3 is nice but very expensive for use in bulk storage at home.
- mycall 7y ago
  What warts do you speak of?
somehnguy 7y ago
Have you tried it? I went from having never used ZFS to loving it (and I guess being one of those zealots) very quickly after setting it up. So simple yet so powerful!
kqr 7y ago
ZFS is the worst filesystem/volume manager, except all others.
twic 7y ago
Eh, i'm waiting for them to rewrite it in Rust.
- paulddraper 7y ago
  Poe's law
- doctor_eval 7y ago
  This is such a meta-comment that I actually LOLd!
rnd0 7y ago
>You can't read any thread that touches on filesystems without the ZFS zealots coming out. Agreed 100%. That's particularly annoying to us desktop users. It took me years to figure out that no, FreeBSD aside, it doesn't bring anything to the table outside of enterprise storage use cases. At least it doesn't bring anything that's worth the hassles (I don't have to export ntfs filesystems before using them on another computer; same for ext4 -and then there's performance).
seized 7y ago
ZFS is like really good snow tires in the winter. You can tell people with other tires how great it is to have really good tires, but they dont believe you until they experience the benefits for themselves. Or put another way, no one "needs" ZFS until they really need it, then they wont live without it ever again. I switched to it after the 7200.11 firmware mess, where the drives reported successful writes but didnt write anything. ZFS would have caught that, my Adaptec card certainly couldnt have and didnt. ZFS to the rescue again a while later when those (now firmware updated) 7200.11 drives started dropping after 15k hours of service. ZFS saved my data when two drives started failing in my RAID5 set at the same time. All the weird minor problems that would cause random issues or performance issues for other file systems like flaky SATA cables, intermittent HBA/backplane ports, etc. ZFS catches them all and informs you. Having been hit by bit rot, corrupted files, corrupted file systems, etc etc before switching, ZFS is fantastic. And there is something great about watching it scrub at >1GB/sec, verifying every single bit of your data.
ksec 7y ago
I had to check the dictionary for the meaning of "hype" a situation in which something is advertised and discussed in newspapers, on television, etc. a lot in order to attract everyone's interest: May be its just me because Morden day usage of "hype" seems to involve and implies a negative meaning, especially in tech. Similar to false advertising. And no one was actively promoting ZFS, they were only very "responsive". And then zealots, I had to reread 226 comments, ran to Cambridge dictionary a person who has very strong opinions about something, and tries to make other people have them too I dont see anyone having strong opinions and force others to have the same. If anything a lot of people are showing not because the love ZFS, but they have been burnt by btrfs.

zionic 7y ago

I have a synology NAS on btrfs. One of the best computer purchases I've ever made.

lostlogin 7y ago
I’ll second this, it’s fantastic. The time it takes to expand when adding a second 16TB is deeply average (8 days) but that’s about it for downsides. It’s the best computer I’ve owned.

m4rtink 7y ago

Hard to standardize on something that can't be maintained in the same place all your other filesystems are in (in the Kernel) for licensing reasons.

rbanffy 7y ago
Only the boot file system drivers need to be in the kernel. As long as there is a stable ABI, it's fine for everything else to be someplace else.
- skissane 7y ago
  > As long as there is a stable ABI, it's fine for everything else to be someplace else. Mainline Linux has a policy against in-kernel ABI stability guarantees. User-space is given ABI stability guarantees, in-kernel code by intention is not. That includes filesystems.

starfallg 7y ago

Btrfs is the only FS I used that resulted in complete FS corruption losing nearly all data on disk, not once, but 3 times. After that, none of the features like compression, snapshots, COW or checksums meant anything to me. I'm much happier with ext4 and xfs on lvm.

RX14 7y ago
It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. One possible cause is distro choice tends to be per person and how up to date said distro keeps it's kernel. But, I'm not sure.
- thfuran 7y ago
  I think the probable cause is that it's not common bugs that cause the corruption but uncommon ones. Most of the time, they work fine. But you really want a stronger guarantee than that out of your filesystem.
  - SEJeff 7y ago
    Historically, the biggest bugs in btrfs were when you came close to filling up the filesystem. For the longest time, you'd get -ENOSPC (no space left) even when you had many Gb of space left due to really bad metadata and block level space usage.
    - blattimwind 7y ago
      Let's not forget about various performance issues which were exacerbated by "low free space" conditions (i.e. after you filled the volume beyond 80 % these started to pop up). A file system that will sometimes go down well into the fractional IOPS range is not very useful. Some of these are fixed by now, though.

tw04 7y ago

Legitimately curious what the ZFS bug was. I’ve not heard of a TFDL bug in zfs for a Loooong time. The reason Synology btrfs is mostly solid is because they refused to ever use the btrfs raid layer. But the second you move to btrfs on LVM you lose a large portion of the supposed benefits. Having used both, never lost data on zfs and I’ve been using it since it was released and have had it save me from silent data corruption. BTRFS hasn’t ever lost me an entire file system, but I’ve definitely lost files.

pizza234 7y ago

Do you have a link to the bug issue? ZFS purpotedly never had any corruption issues on release versions, so that makes it a really interesting case.