6 ms·

I'd be curious to hear about the issues you've had with zfs. I'm running it on a fileserver in my house, and it's been solid for a few years, but my use isn't v

I'd be curious to hear about the issues you've had with zfs. I'm running it on a fileserver in my house, and it's been solid for a few years, but my use isn't very demanding. I'd like to know what problems I should be keeping an eye out for (aside from the usual stuff like drive failures, etc.).

tracker1 10y ago
It wasn't ZFS perse... the software I was using wouldn't properly enable the drives configured as hot spares... that and a few errors (I didn't use ECC), and a couple bad drives, and everything (11TB of data) lost. All the really important stuff was well backed up, as was my audio library. I never did re-rip all my DVDs & BluRays again since that happened.
- zippergz 10y ago
  Good info, thanks. I'll admit, I'm pretty terrified of having to replace one of the drives in my array. I've replaced dozens or maybe even hundreds of drives in Sun, SGI and DEC storage arrays over the years, but I have a lot less confidence in my homebrew setup. Fortunately, I have regular complete backups of everything there, so even if I lost the whole array, all I'd really lose is the time involved in fixing it, but that's still painful.
  - tracker1 10y ago
    I felt the same way, which was why I went Raid-Z2 (dual parity) with two hot spares configured... even had a spare controller next to the box. Still, lost it all.
    - brokenmachine 10y ago
      Damn, how did that happen? When the first drive failed, why didn't the hot spare get activated? What software was it that didn't work properly? This worries me because Raid-Z2 with two hot spares is exactly the system that I was planning to set up.
      - tracker1 10y ago
        The FreeNAS version I was using had the bug wrt hot spares... and I didn't use ECC... The errors also weren't clear which drive was failing... was 0 in some places 1 in others, and it turns out it was both. The LSI board I was using apparently had some buggy issues as well. That combined with using Seagate 7200 rpm drives in a year that was particularly bad (an offline storage company mentioned them in a blog article iirc). just a bad chain of events. The drives when I pulled out and tested them individually after the entire array crashed, about 7 out of 12 had significant issues, and of the other 5, 2 more developed errors shortly after (I only used them as scratch/project drives). Honestly, today, I'd probably get another Synology box, a bit more expensive, but been so much less hassle. I had upgraded my 4-drive (2010 plus model iirc) synology box to 4TB wd red drives, and been running that ever since. I don't think I'd ever do a homebrew NAS on a single machine ever again. If I had to do something in a company, would probably lean towards distributed file systems for that level of redundancy.
        colejohnson66 10y ago
        Were there Baracuda 7200.11? That firmware version was notorious for corruption
        tracker1 10y ago
        IIRC, yes... horrible drives.
        lmm 10y ago
        Were you not scrubbing at all? (Blaming FreeNAS rather than you if so - but if one's doing it by hand following the ZFS HOWTOs it's definitely something that's impressed on one to set up).
        tracker1 10y ago
        When I use a NAS software, that's meant to be a NAS, I expect it to do maintenance that's required. I don't expect to have to be an expert in ZFS. By the same note, I don't expect to have to know what the storage and growth configurations of AWS RDS instances either. My Synology NAS device has been in service for around 6 years now... I upgraded the drives from 4x1tb to 4x4TB when the freenas server bit the dust... and it's still going strong... I don't have to think about it, it just sits there and runs.... A single-purpose software like this should do its' job. The end.
- hvidgaard 10y ago
  I've lost my movie collection to a different issue, and I never got around to re-rip it again. But I have promised myself that if I have to rip it again, I will do it script based, such that backing the script up, re-ripping will be a matter of installing the same software, and running the script.
  - tracker1 10y ago
    true enough... At least today I'd get the benefit of h.265 and use roughly 1/4 the space, and probably go 1080p for the BR rips instead of 720p.
aroch 10y ago
ZFS pool IO performance drops like a rock off a very tall cliff once the pool is in the 80%+ full range. So for every TB of usable pool space, you need to over-provision by 200-300GB. This is with current gene enterprise SSDs mind you. Even worse things happen when using spinning rust -- like ZFS dropping drives from the pool because IO has degraded so much it thinks theres a bad drive. It turns out that some raid controllers (Including the very popular LSI variety) + certain kernel versions result in pools and drives being dropped from the array for no discernible reason. Nothing [useful] to logs, just reboot and pray Non-ECC RAM is just asking for trouble in any circumstance where you care about both data integrity and IOPS. Bitflips can and will get you, now they won't wind up hosing your entire pool (most likely) like some apocryphal stories suggest. But, you'll get bad data replicated and the possibility of borking some other sector during the 'recovery' process from the bitflip. When you're in a large enough pool (PB-scale) this becomes even more painful...
- maccam94 10y ago
  I believe adding a ZIL device alleviates the low free space performance issue.
  - aroch 10y ago
    Adding a ZIL device is equivalent to over-provisioning, but with performance impacts in high performance situations
  - rdtsc 10y ago
    From what I understand a ZIL helps with workloads which need to fsync a lot (say databases) and also NFS (due to COMMIT commands issued by clients).
- ahl 10y ago
  100% true about the performance cliff, though at Delphix we worked a bunch to raise performance before the cliff and to push out that cliff. All COW filesystems filesystems (and I'll throw SSD FTLs into there as well) require over-provisioning for consistent performance. And it's also a fair point that for enterprise storage running at 80%-90% of capacity is a reasonable restriction whereas the drive on my laptop is always basically %99 full. It will be interesting to see how APFS addresses this since I'd guess that most of their users look more like my laptop than my file server.
rodgerd 10y ago
I've been running ZFS in prod since 2008 (on Solaris). Bear in mind that was not long after the "silent data corruption" problem that bit Joyent, but more quotodien problems such as: * no reserved space on the disk means you can fill it up and then not be able to delete files or extend it because there was no space for COW operations. Sun's answer? Delete the filesystem, recreate and restore from backup. * zpool management is a pain in the arse generally, and the zpool/zfs distinction is pretty clearly designed on the assumption you'll have sysadmins paying a lot of attention to the low lever FS and pool management. Imagine Steve Jobs being told "your disk filled up and now you need to reformat it". Apple didn't reject ZFS because (whatever some embittered Sun blogger says) of NIH. They rejected it because it wasn't fit for purpose as a consumer desktop filesystem.
- ScottBurson 10y ago
  There's also the problem that you can't grow a raidz vdev by adding disks. I can just imagine Apple support trying to explain that one.
  - lmm 10y ago
    In Solaris ZFS you can now AIUI - it's just that hasn't made it into the open-source version.
  - xenophonf 10y ago
    That's true, but you can grow a pool by progressively replacing smaller disks with larger disks.
- hvidgaard 10y ago
  But surely, it would have been a simpler task to modify ZFS to support these things. I mean, the full disk issue is relatively straight forward to fix. And I have a hard time imaging that other issues would be harder than writing your own from scratch.
  - rodgerd 10y ago
    > But surely, it would have been a simpler task to modify ZFS to support these things. The first time I encountered the disk full problem was 2009. There were emails from Sun engineers saying "this is a dumb situation, we should have a reserved block of storage to avoid this" from 2006 and nothing had been done. Why would anyone in Apple believe if was going to get better if Sun's response to paying customers was "reformat and rebuild" for at least three years? There was no way of fixing the limitations around vdevs, like not being able to grow existing arrays, not mixing block sizes without a big perfomance hit, and so on, without a huge overhaul of some of the fundamentals. ZFS was, like a lot of Sun technologies, built on assumptions from a world where machines are scarce, have a high ratio of admins:machines, and many users per machines. It's excellent at meeting its design goals, but there is no way I can imagine Apple shipping it when they'd be telling people, "want to grow your Mac Pro's array? Rebuild it. Want to reclaim space on a full system? Rebuild it. Want avoid dropping off a performance cliff? Never drop below 20% free space." (As an aside, I've been told by Sun engineers around the same 2008-2009 era that dependency resolution in packagme managers for distros proved Linux was a shit toy for morons because "real" OSes assumed skilled engineers who would be inspecting patches package-by-package so they knew exactly what would be on the system. Even recently, with Solaris 11 shipping with a package manager, I've had Sun veterans complain that "Solaris is dead" because it's "too much like Linux". Sun appears to have fostered, at least in areas I've dealt with, a culture that's completely out-of-step with how computing works this century. I doubt it would have taken too many of those sorts of conversations to convince people at Apple that Sun just didn't get Apple's "just works" vision.)

tracker1 10y ago

It wasn't ZFS perse... the software I was using wouldn't properly enable the drives configured as hot spares... that and a few errors (I didn't use ECC), and a couple bad drives, and everything (11TB of data) lost. All the really important stuff was well backed up, as was my audio library. I never did re-rip all my DVDs & BluRays again since that happened.

zippergz 10y ago
Good info, thanks. I'll admit, I'm pretty terrified of having to replace one of the drives in my array. I've replaced dozens or maybe even hundreds of drives in Sun, SGI and DEC storage arrays over the years, but I have a lot less confidence in my homebrew setup. Fortunately, I have regular complete backups of everything there, so even if I lost the whole array, all I'd really lose is the time involved in fixing it, but that's still painful.
- tracker1 10y ago
  I felt the same way, which was why I went Raid-Z2 (dual parity) with two hot spares configured... even had a spare controller next to the box. Still, lost it all.
  - brokenmachine 10y ago
    Damn, how did that happen? When the first drive failed, why didn't the hot spare get activated? What software was it that didn't work properly? This worries me because Raid-Z2 with two hot spares is exactly the system that I was planning to set up.
    - tracker1 10y ago
      The FreeNAS version I was using had the bug wrt hot spares... and I didn't use ECC... The errors also weren't clear which drive was failing... was 0 in some places 1 in others, and it turns out it was both. The LSI board I was using apparently had some buggy issues as well. That combined with using Seagate 7200 rpm drives in a year that was particularly bad (an offline storage company mentioned them in a blog article iirc). just a bad chain of events. The drives when I pulled out and tested them individually after the entire array crashed, about 7 out of 12 had significant issues, and of the other 5, 2 more developed errors shortly after (I only used them as scratch/project drives). Honestly, today, I'd probably get another Synology box, a bit more expensive, but been so much less hassle. I had upgraded my 4-drive (2010 plus model iirc) synology box to 4TB wd red drives, and been running that ever since. I don't think I'd ever do a homebrew NAS on a single machine ever again. If I had to do something in a company, would probably lean towards distributed file systems for that level of redundancy.
      - colejohnson66 10y ago
        Were there Baracuda 7200.11? That firmware version was notorious for corruption
        tracker1 10y ago
        IIRC, yes... horrible drives.
      - lmm 10y ago
        Were you not scrubbing at all? (Blaming FreeNAS rather than you if so - but if one's doing it by hand following the ZFS HOWTOs it's definitely something that's impressed on one to set up).
        tracker1 10y ago
        When I use a NAS software, that's meant to be a NAS, I expect it to do maintenance that's required. I don't expect to have to be an expert in ZFS. By the same note, I don't expect to have to know what the storage and growth configurations of AWS RDS instances either. My Synology NAS device has been in service for around 6 years now... I upgraded the drives from 4x1tb to 4x4TB when the freenas server bit the dust... and it's still going strong... I don't have to think about it, it just sits there and runs.... A single-purpose software like this should do its' job. The end.
hvidgaard 10y ago
I've lost my movie collection to a different issue, and I never got around to re-rip it again. But I have promised myself that if I have to rip it again, I will do it script based, such that backing the script up, re-ripping will be a matter of installing the same software, and running the script.
- tracker1 10y ago
  true enough... At least today I'd get the benefit of h.265 and use roughly 1/4 the space, and probably go 1080p for the BR rips instead of 720p.

aroch 10y ago

ZFS pool IO performance drops like a rock off a very tall cliff once the pool is in the 80%+ full range. So for every TB of usable pool space, you need to over-provision by 200-300GB. This is with current gene enterprise SSDs mind you. Even worse things happen when using spinning rust -- like ZFS dropping drives from the pool because IO has degraded so much it thinks theres a bad drive. It turns out that some raid controllers (Including the very popular LSI variety) + certain kernel versions result in pools and drives being dropped from the array for no discernible reason. Nothing [useful] to logs, just reboot and pray Non-ECC RAM is just asking for trouble in any circumstance where you care about both data integrity and IOPS. Bitflips can and will get you, now they won't wind up hosing your entire pool (most likely) like some apocryphal stories suggest. But, you'll get bad data replicated and the possibility of borking some other sector during the 'recovery' process from the bitflip. When you're in a large enough pool (PB-scale) this becomes even more painful...

maccam94 10y ago
I believe adding a ZIL device alleviates the low free space performance issue.
- aroch 10y ago
  Adding a ZIL device is equivalent to over-provisioning, but with performance impacts in high performance situations
- rdtsc 10y ago
  From what I understand a ZIL helps with workloads which need to fsync a lot (say databases) and also NFS (due to COMMIT commands issued by clients).
ahl 10y ago
100% true about the performance cliff, though at Delphix we worked a bunch to raise performance before the cliff and to push out that cliff. All COW filesystems filesystems (and I'll throw SSD FTLs into there as well) require over-provisioning for consistent performance. And it's also a fair point that for enterprise storage running at 80%-90% of capacity is a reasonable restriction whereas the drive on my laptop is always basically %99 full. It will be interesting to see how APFS addresses this since I'd guess that most of their users look more like my laptop than my file server.

rodgerd 10y ago

I've been running ZFS in prod since 2008 (on Solaris). Bear in mind that was not long after the "silent data corruption" problem that bit Joyent, but more quotodien problems such as: * no reserved space on the disk means you can fill it up and then not be able to delete files or extend it because there was no space for COW operations. Sun's answer? Delete the filesystem, recreate and restore from backup. * zpool management is a pain in the arse generally, and the zpool/zfs distinction is pretty clearly designed on the assumption you'll have sysadmins paying a lot of attention to the low lever FS and pool management. Imagine Steve Jobs being told "your disk filled up and now you need to reformat it". Apple didn't reject ZFS because (whatever some embittered Sun blogger says) of NIH. They rejected it because it wasn't fit for purpose as a consumer desktop filesystem.

ScottBurson 10y ago
There's also the problem that you can't grow a raidz vdev by adding disks. I can just imagine Apple support trying to explain that one.
- lmm 10y ago
  In Solaris ZFS you can now AIUI - it's just that hasn't made it into the open-source version.
- xenophonf 10y ago
  That's true, but you can grow a pool by progressively replacing smaller disks with larger disks.
hvidgaard 10y ago
But surely, it would have been a simpler task to modify ZFS to support these things. I mean, the full disk issue is relatively straight forward to fix. And I have a hard time imaging that other issues would be harder than writing your own from scratch.
- rodgerd 10y ago
  > But surely, it would have been a simpler task to modify ZFS to support these things. The first time I encountered the disk full problem was 2009. There were emails from Sun engineers saying "this is a dumb situation, we should have a reserved block of storage to avoid this" from 2006 and nothing had been done. Why would anyone in Apple believe if was going to get better if Sun's response to paying customers was "reformat and rebuild" for at least three years? There was no way of fixing the limitations around vdevs, like not being able to grow existing arrays, not mixing block sizes without a big perfomance hit, and so on, without a huge overhaul of some of the fundamentals. ZFS was, like a lot of Sun technologies, built on assumptions from a world where machines are scarce, have a high ratio of admins:machines, and many users per machines. It's excellent at meeting its design goals, but there is no way I can imagine Apple shipping it when they'd be telling people, "want to grow your Mac Pro's array? Rebuild it. Want to reclaim space on a full system? Rebuild it. Want avoid dropping off a performance cliff? Never drop below 20% free space." (As an aside, I've been told by Sun engineers around the same 2008-2009 era that dependency resolution in packagme managers for distros proved Linux was a shit toy for morons because "real" OSes assumed skilled engineers who would be inspecting patches package-by-package so they knew exactly what would be on the system. Even recently, with Solaris 11 shipping with a package manager, I've had Sun veterans complain that "Solaris is dead" because it's "too much like Linux". Sun appears to have fostered, at least in areas I've dealt with, a culture that's completely out-of-step with how computing works this century. I doubt it would have taken too many of those sorts of conversations to convince people at Apple that Sun just didn't get Apple's "just works" vision.)