14 ms·

What's the stability of ZFS on linux like these days? Anyone have any positive / negative experiences to share?

aeorgnoieang 10y ago
I've been running two very-lightly-used sets of pools on two old desktops for ... about a year and a half now actually and everything's gone swimmingly. Nor have I read of anything bad happening to anyone that didn't involve not having backups. All in all, lots of people are running it, lots of people seem to be having almost entirely positive experiences with it, so it seems like it's very stable.
- agumonkey 10y ago
  Could you give machine specs ?
  - barrkel 10y ago
    $ head -1 /proc/meminfo MemTotal: 4046880 kB $ grep model.name /proc/cpuinfo | head -1 model name : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz $ sudo zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 36.2T 27.2T 9.09T - 10% 74% 1.00x ONLINE - That's 10x 4TB WD Red drives in raidz2 configuration. I've been running ZFS for the past 6 years, a little over 2 years on ZoL, and 4 years before that on Nexenta (Ubuntu userland with OpenSolaris kernel). I've had 5 HDD failures over the past few years, never lost any data. I've found ZoL to have less jitter when streaming video files in particular over SMB than the Solaris implementation did. ZoL is tighter on memory though. raidz2 isn't particularly fast for random I/O (since it acts like it has a single spindle), but I don't do much random I/O. It's mostly video and local backup cache. I'll be thinking about expanding my current setup soon enough. I'll probably be aiming for a raid10 setup for better random access, and of course the new system will have more RAM, and ECC.
    - agumonkey 10y ago
      Thanks. Beefier cpu than the beefier I got here, but RAM is a great hint that for personal needs ZFS can run on normal machines. Interesting that you had no issues without ECC in the first place. Some ZFS boards are pretty against trying ZFS without.
      - rincebrain 10y ago
        ZFS boards really encourage you to use ECC because one of the nominal features of ZFS, "end-to-end checksumming", is not quite as useful if you have a point between ends where it won't notice silent data corruption (e.g. a bad stick of non-ECC RAM). ZFS will notice, depending on _when_ the munging happened (if it was mangled after it got to RAM but before ZFS checksummed it, it obviously can't), but that's (primarily) why ZFS boards often tell you not to do that.
        erkkie 10y ago
        Not only that, but I believe a bad checksum can also cause a cascade to "fix the data", causing further corruption.
        the_why_of_y 10y ago
        For that to happen, you would need a bad checksum and the corresponding data block to be corrupted in such a way that the corrupted data block checksums to the corrupted checksum. Given that the checksum is 256-bit, the scariest I'd call that is "possible in theory".
      - mioelnir 10y ago
        > Interesting that you had no issues without ECC in the first place. Some ZFS boards are pretty against trying ZFS without. ZFS without ECC is no better or worse than any other filesystem without ECC, in that silent data corruption is possible. It is for this reason just a lot worse than ZFS with ECC, introducing a fault condition normally not seen on ZFS. But that fault condition does not go away because you chose to skip ZFS due to no ECC. And you will get a lot of other problems on top that ZFS would have helped against, even without ECC. Lastly, you'll end up back in the horrible land of static partitioning schemes with questionable tooling. Why suffer through that if you do not have to?
  - doublerebel 10y ago
    ZFS can run on pretty low specs unless it's being pushed hard. I use a low-power AMD Sempron to keep things quiet. These specs are enough for all my backup and video streaming needs: $ head -1 /proc/meminfo MemTotal: 1967672 kB $ grep model.name /proc/cpuinfo | head -1 model name: AMD Sempron(tm) 145 Processor
    - gonzo 10y ago
      The in-development version of pfsense allows installs to ZFS, and I've installed onto a 2c atom with 2GB ram and a small SSD. But it's not as performant as UFS on the same hardware.
- saurik 10y ago
  > Nor have I read of anything bad happening to anyone that didn't involve not having backups. This should be true of any storage technology, no matter how unreliable (and thereby undermines the anecdote).
  - devplusops 10y ago
    The trouble is it depends how you define 'backup'. E.g. if you are using zfs send/receive to do backups, and there's a bug in zfs send/receive.... Some people advise making sure you backup using different techniques/strategies/software, but in practice that gets quite difficult to manage, and you quickly end up going down a very deep rabbit hold on your quest for independent backup strategies. Consider this example: I'm aware of at least one instance where a faulty tape drive damaged the tapes used by that drive - in a a way that caused those tapes to damage other drives in the same way. The damage spread like a virus. Unless you have "blue" tapes, and "yellow" tapes, and "blue" drives, and "yellow drives", an issue like that won't be contained. But you do the combinatorics on all the kinds of issues like that which need to be addressed, and quickly you'll end up spending your whole life backing up data. Which will sort of solve the problem, because you won't have time to create any data which needs to be backed up.
- tobias3 10y ago
  As long as you don't backup with send/receive. ("Silently corrupted file in snapshots after send/receive" https://github.com/zfsonlinux/zfs/issues/4809 https://github.com/zfsonlinux/zfs/issues/4809)
  - rincebrain 10y ago
    That's a bug in OpenZFS, to be fair, not ZoL-specific, though 0.6.5 not having a fix merged for a while is slightly surprising. Just grab https://github.com/zfsonlinux/zfs/pull/4833 https://github.com/zfsonlinux/zfs/pull/4833 and send/recv in safety. ;)
kellyjprice 10y ago
I've been using it casually as my /home for ~2 years. The most stress it gets is torrenting movies to it. But, there's been no problems. Setting it up on 3 drives was dead simple.
rcthompson 10y ago
It's quite good. I've been running ZFS on my Ubuntu server for about a year now. It's been quite solid. It detected and corrected some corruption resulting from overheating due to fans caked with dust. I've recently switched over to root-on-ZFS and it's continued to run well.
badmadrad 10y ago
I have been using it on Ubuntu 16.04 and before that with Ubuntu 15.10 for about a year. Good news is its a first class citizen on Ubuntu now. I have have great success with it on my home server and have plans to try for some prod systems I'm building out. https://insights.ubuntu.com/2016/02/16/zfs-is-the-fs-for-containers-in-ubuntu-16-04/ https://insights.ubuntu.com/2016/02/16/zfs-is-the-fs-for-con... https://wiki.ubuntu.com/ZFS https://wiki.ubuntu.com/ZFS
drewg123 10y ago
I've been running it on a machine I built in 2011, and it seems to work quite well. It started out as whatever Ubuntu was available in August of 2011, then 12.04, then 14.04. I have 3 3x 3TB Hitachi Deskstars in a raids, with a crucial M4-CT128M4 as a combined rootfs and l2arc (the ssd was an upgrade from an older ssd when I moved from 12.04 to 14.04). The machine has a Xeon e3-1270 CPU, and 16GB of ECC RAM. I used it as a workstation (now my wife does), and it has been as a home media / tv server / NAS for its entire lifetime. 0 problems so far (knock on wood), and it has made a move from the east coast to California, and then back to the east coast again.
- e40 10y ago
  What chassis are you using? And do you like it? I'm in the market. Thanks.
  - drewg123 10y ago
    I like the chassis, but the build was ~5 years ago, and when I built a new machine ~1 year ago, it was not available. My current machine uses a Fractal Design Define R5 Black Silent ATX Midtower case. That one is quite nice. The one gotcha that I had is that it is a bit wider than I was used to, which means that it just barely fits in the computer holder accessory for my ikea desk.
doublerebel 10y ago
I've been running it for over 8 years on Arch Linux using the AUR packages, from FUSE through to the kernel driver. I run a RAIDZ2 pool that is now upgraded to ~4TB capacity from 4 2TB disks. I have had no known data loss. I did once lose my (non-ZFS) boot drive to hardware failure and the zpool mounted right up with a fresh installation.
wspeirs 10y ago
We run ZFS at my company, Datto, across hundreds of machines, including devices out in the field and servers in our data centers. It has a few warts, but overall very stable and the feature set has allowed us to deliver world-class features to our customers.
- NovaS1X 10y ago
  What are the few warts you encounter? I'm a Sysadmin/Engineer at a VFX company. We have hundreds of machines, render, and servers all running Linux but we're typically on ext4 with the notable exception of our Isilon cluster. What does your environment look like?
  - e40 10y ago
    One that I've heard often repeated (paging user rsync) is that you should never fill your filesystems past 80% (or so). Things get slow otherwise.
    - oofabz 10y ago
      ZFS certainly slows down as it fills up, but only enough to affect high-IO, write-intensive loads like databases. A light-weight application like a home or office file server can still saturate gigabit ethernet even while 99% full.
    - NovaS1X 10y ago
      This is also true of other filesystems however. I don't have experience with high IO on ZFS at those rumored 80%+ capacities however once a core filer fills up to 90% you start seeing the tickets rolling in and application performance tank. 80% if usually fine however. I wonder to what degree the performance degradation is on OpenZFS.
      - throwaway7767 10y ago
        The problem is much more severe on ZFS and other CoW filesystems, hence the common warning. Filling an ext4 filesystem up to 80%+ and then removing files might increase fragmentation a bit but it'll not have that much effect on performance. On ZFS, filling the fileystem up once can kill performance permanently on that pool.
lfittl 10y ago
We've evaluated it for our cloud database service, but ended up going back to ext4 due to memory management bugs. In particular this one: https://github.com/zfsonlinux/zfs/issues/3645 https://github.com/zfsonlinux/zfs/issues/3645
- kim0 10y ago
  This kind of thing is what's better on platforms like freebsd
- Zancarius 10y ago
  I ran into a bug [1] that appeared to be a deadlock triggered by rsync of a relatively modest directory with tons of small files. The only thing that fixed it was `spl_taskq_thread_dynamic=0` to stop the dynamic spawning of ZFS-related kernel threads. Rather than a memory management issue, it'd stall the copy and peg my CPUs indefinitely. I suspect it's fixed in 0.7.0 based on some of the other related bugs I've run into since, but I've been reluctant to upgrade as of yet. Otherwise, ZFS on a home file server has been great. [1] https://github.com/zfsonlinux/zfs/issues/3808 https://github.com/zfsonlinux/zfs/issues/3808
  - tobias3 10y ago
    The Ubuntu ZFS module sets spl_taskq_thread_dynamic to zero by default. My problem was that the large number threads with spl_taskq_thread_dynamic=0 cause it to OOM, at one point it had an OOM at mount (with 32GB RAM). When I set spl_taskq_thread_dynamic=1 I had the same deadlock issue (That's where I stopped trying to use it and went back to FreeBSD).
    - Annatar 10y ago
      The fix is not to work around the out of memory killer in Linux, but to turn off memory overcommit altogether: the system should not lie to applications that it has more virtual memory than is actually available, because that is extremely detrimental to correctness of operation. illumos based systems, for instance, never lie about such things.
funkaster 10y ago
I've been running it on my laptop (chromebook pixel 2) for several months so far and it's great. I was waiting for this to be merged for a long time. Right now I have /home as a zpool than runs from an SDCard on my computer. The SD Card is a 64Gb card (the most expensive/fast I could find) and for my workload, it works just amazing: this setup leaves me with 64Gb (a bit less) for / and 64Gb for /home. The ability to cd into snapshots is just great and a time saver. So far, nothing wrong/bad with this.

I've been running two very-lightly-used sets of pools on two old desktops for ... about a year and a half now actually and everything's gone swimmingly. Nor have I read of anything bad happening to anyone that didn't involve not having backups. All in all, lots of people are running it, lots of people seem to be having almost entirely positive experiences with it, so it seems like it's very stable.

agumonkey 10y ago
Could you give machine specs ?
- barrkel 10y ago
  $ head -1 /proc/meminfo MemTotal: 4046880 kB $ grep model.name /proc/cpuinfo | head -1 model name : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz $ sudo zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 36.2T 27.2T 9.09T - 10% 74% 1.00x ONLINE - That's 10x 4TB WD Red drives in raidz2 configuration. I've been running ZFS for the past 6 years, a little over 2 years on ZoL, and 4 years before that on Nexenta (Ubuntu userland with OpenSolaris kernel). I've had 5 HDD failures over the past few years, never lost any data. I've found ZoL to have less jitter when streaming video files in particular over SMB than the Solaris implementation did. ZoL is tighter on memory though. raidz2 isn't particularly fast for random I/O (since it acts like it has a single spindle), but I don't do much random I/O. It's mostly video and local backup cache. I'll be thinking about expanding my current setup soon enough. I'll probably be aiming for a raid10 setup for better random access, and of course the new system will have more RAM, and ECC.
  - agumonkey 10y ago
    Thanks. Beefier cpu than the beefier I got here, but RAM is a great hint that for personal needs ZFS can run on normal machines. Interesting that you had no issues without ECC in the first place. Some ZFS boards are pretty against trying ZFS without.
    - rincebrain 10y ago
      ZFS boards really encourage you to use ECC because one of the nominal features of ZFS, "end-to-end checksumming", is not quite as useful if you have a point between ends where it won't notice silent data corruption (e.g. a bad stick of non-ECC RAM). ZFS will notice, depending on _when_ the munging happened (if it was mangled after it got to RAM but before ZFS checksummed it, it obviously can't), but that's (primarily) why ZFS boards often tell you not to do that.
      - erkkie 10y ago
        Not only that, but I believe a bad checksum can also cause a cascade to "fix the data", causing further corruption.
        the_why_of_y 10y ago
        For that to happen, you would need a bad checksum and the corresponding data block to be corrupted in such a way that the corrupted data block checksums to the corrupted checksum. Given that the checksum is 256-bit, the scariest I'd call that is "possible in theory".
    - mioelnir 10y ago
      > Interesting that you had no issues without ECC in the first place. Some ZFS boards are pretty against trying ZFS without. ZFS without ECC is no better or worse than any other filesystem without ECC, in that silent data corruption is possible. It is for this reason just a lot worse than ZFS with ECC, introducing a fault condition normally not seen on ZFS. But that fault condition does not go away because you chose to skip ZFS due to no ECC. And you will get a lot of other problems on top that ZFS would have helped against, even without ECC. Lastly, you'll end up back in the horrible land of static partitioning schemes with questionable tooling. Why suffer through that if you do not have to?
- doublerebel 10y ago
  ZFS can run on pretty low specs unless it's being pushed hard. I use a low-power AMD Sempron to keep things quiet. These specs are enough for all my backup and video streaming needs: $ head -1 /proc/meminfo MemTotal: 1967672 kB $ grep model.name /proc/cpuinfo | head -1 model name: AMD Sempron(tm) 145 Processor
  - gonzo 10y ago
    The in-development version of pfsense allows installs to ZFS, and I've installed onto a 2c atom with 2GB ram and a small SSD. But it's not as performant as UFS on the same hardware.
saurik 10y ago
> Nor have I read of anything bad happening to anyone that didn't involve not having backups. This should be true of any storage technology, no matter how unreliable (and thereby undermines the anecdote).
- devplusops 10y ago
  The trouble is it depends how you define 'backup'. E.g. if you are using zfs send/receive to do backups, and there's a bug in zfs send/receive.... Some people advise making sure you backup using different techniques/strategies/software, but in practice that gets quite difficult to manage, and you quickly end up going down a very deep rabbit hold on your quest for independent backup strategies. Consider this example: I'm aware of at least one instance where a faulty tape drive damaged the tapes used by that drive - in a a way that caused those tapes to damage other drives in the same way. The damage spread like a virus. Unless you have "blue" tapes, and "yellow" tapes, and "blue" drives, and "yellow drives", an issue like that won't be contained. But you do the combinatorics on all the kinds of issues like that which need to be addressed, and quickly you'll end up spending your whole life backing up data. Which will sort of solve the problem, because you won't have time to create any data which needs to be backed up.
tobias3 10y ago
As long as you don't backup with send/receive. ("Silently corrupted file in snapshots after send/receive" https://github.com/zfsonlinux/zfs/issues/4809 https://github.com/zfsonlinux/zfs/issues/4809)
- rincebrain 10y ago
  That's a bug in OpenZFS, to be fair, not ZoL-specific, though 0.6.5 not having a fix merged for a while is slightly surprising. Just grab https://github.com/zfsonlinux/zfs/pull/4833 https://github.com/zfsonlinux/zfs/pull/4833 and send/recv in safety. ;)

kellyjprice 10y ago

I've been using it casually as my /home for ~2 years. The most stress it gets is torrenting movies to it. But, there's been no problems. Setting it up on 3 drives was dead simple.

rcthompson 10y ago

It's quite good. I've been running ZFS on my Ubuntu server for about a year now. It's been quite solid. It detected and corrected some corruption resulting from overheating due to fans caked with dust. I've recently switched over to root-on-ZFS and it's continued to run well.

badmadrad 10y ago

I have been using it on Ubuntu 16.04 and before that with Ubuntu 15.10 for about a year. Good news is its a first class citizen on Ubuntu now. I have have great success with it on my home server and have plans to try for some prod systems I'm building out. https://insights.ubuntu.com/2016/02/16/zfs-is-the-fs-for-containers-in-ubuntu-16-04/ https://insights.ubuntu.com/2016/02/16/zfs-is-the-fs-for-con... https://wiki.ubuntu.com/ZFS https://wiki.ubuntu.com/ZFS

drewg123 10y ago

I've been running it on a machine I built in 2011, and it seems to work quite well. It started out as whatever Ubuntu was available in August of 2011, then 12.04, then 14.04. I have 3 3x 3TB Hitachi Deskstars in a raids, with a crucial M4-CT128M4 as a combined rootfs and l2arc (the ssd was an upgrade from an older ssd when I moved from 12.04 to 14.04). The machine has a Xeon e3-1270 CPU, and 16GB of ECC RAM. I used it as a workstation (now my wife does), and it has been as a home media / tv server / NAS for its entire lifetime. 0 problems so far (knock on wood), and it has made a move from the east coast to California, and then back to the east coast again.

e40 10y ago
What chassis are you using? And do you like it? I'm in the market. Thanks.
- drewg123 10y ago
  I like the chassis, but the build was ~5 years ago, and when I built a new machine ~1 year ago, it was not available. My current machine uses a Fractal Design Define R5 Black Silent ATX Midtower case. That one is quite nice. The one gotcha that I had is that it is a bit wider than I was used to, which means that it just barely fits in the computer holder accessory for my ikea desk.

doublerebel 10y ago

I've been running it for over 8 years on Arch Linux using the AUR packages, from FUSE through to the kernel driver. I run a RAIDZ2 pool that is now upgraded to ~4TB capacity from 4 2TB disks. I have had no known data loss. I did once lose my (non-ZFS) boot drive to hardware failure and the zpool mounted right up with a fresh installation.

wspeirs 10y ago

We run ZFS at my company, Datto, across hundreds of machines, including devices out in the field and servers in our data centers. It has a few warts, but overall very stable and the feature set has allowed us to deliver world-class features to our customers.

NovaS1X 10y ago
What are the few warts you encounter? I'm a Sysadmin/Engineer at a VFX company. We have hundreds of machines, render, and servers all running Linux but we're typically on ext4 with the notable exception of our Isilon cluster. What does your environment look like?
- e40 10y ago
  One that I've heard often repeated (paging user rsync) is that you should never fill your filesystems past 80% (or so). Things get slow otherwise.
  - oofabz 10y ago
    ZFS certainly slows down as it fills up, but only enough to affect high-IO, write-intensive loads like databases. A light-weight application like a home or office file server can still saturate gigabit ethernet even while 99% full.
  - NovaS1X 10y ago
    This is also true of other filesystems however. I don't have experience with high IO on ZFS at those rumored 80%+ capacities however once a core filer fills up to 90% you start seeing the tickets rolling in and application performance tank. 80% if usually fine however. I wonder to what degree the performance degradation is on OpenZFS.
    - throwaway7767 10y ago
      The problem is much more severe on ZFS and other CoW filesystems, hence the common warning. Filling an ext4 filesystem up to 80%+ and then removing files might increase fragmentation a bit but it'll not have that much effect on performance. On ZFS, filling the fileystem up once can kill performance permanently on that pool.

lfittl 10y ago

We've evaluated it for our cloud database service, but ended up going back to ext4 due to memory management bugs. In particular this one: https://github.com/zfsonlinux/zfs/issues/3645 https://github.com/zfsonlinux/zfs/issues/3645

kim0 10y ago
This kind of thing is what's better on platforms like freebsd
Zancarius 10y ago
I ran into a bug [1] that appeared to be a deadlock triggered by rsync of a relatively modest directory with tons of small files. The only thing that fixed it was `spl_taskq_thread_dynamic=0` to stop the dynamic spawning of ZFS-related kernel threads. Rather than a memory management issue, it'd stall the copy and peg my CPUs indefinitely. I suspect it's fixed in 0.7.0 based on some of the other related bugs I've run into since, but I've been reluctant to upgrade as of yet. Otherwise, ZFS on a home file server has been great. [1] https://github.com/zfsonlinux/zfs/issues/3808 https://github.com/zfsonlinux/zfs/issues/3808
- tobias3 10y ago
  The Ubuntu ZFS module sets spl_taskq_thread_dynamic to zero by default. My problem was that the large number threads with spl_taskq_thread_dynamic=0 cause it to OOM, at one point it had an OOM at mount (with 32GB RAM). When I set spl_taskq_thread_dynamic=1 I had the same deadlock issue (That's where I stopped trying to use it and went back to FreeBSD).
  - Annatar 10y ago
    The fix is not to work around the out of memory killer in Linux, but to turn off memory overcommit altogether: the system should not lie to applications that it has more virtual memory than is actually available, because that is extremely detrimental to correctness of operation. illumos based systems, for instance, never lie about such things.

funkaster 10y ago

I've been running it on my laptop (chromebook pixel 2) for several months so far and it's great. I was waiting for this to be merged for a long time. Right now I have /home as a zpool than runs from an SDCard on my computer. The SD Card is a 64Gb card (the most expensive/fast I could find) and for my workload, it works just amazing: this setup leaves me with 64Gb (a bit less) for / and 64Gb for /home. The ability to cd into snapshots is just great and a time saver. So far, nothing wrong/bad with this.