6 ms·

For years I’m running home server. Mostly for storage plus some side projects. I only had a little time for server maintenance and I ended up redoing the whole

For years I’m running home server. Mostly for storage plus some side projects. I only had a little time for server maintenance and I ended up redoing the whole server every time a disk failed. I was using just regular desktop grade disks so that happened every 18 months or so. This all changed when I started using ZFS. Not only it has support for raidz and mirroring (which I could get with LVM too) but it is tweekable and tunable easily. Plus commands like _zpool status_ will give you a great overview of the health of the array in no time. It might seem like nothing but it makes all the difference for me. I can recommand for everybody running a (home) server. It will save you lots of time.

emilburzo 6y ago
Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening. Also, 18 months sounds kind of short, I usually get around 5 years of use from hard drives. What brand are you using? I highly recommend checking out the quarterly Backblaze HDD failure reports. Source: been running a server (read: consumer headless desktop) for years without issues.
- redflame7 6y ago
  More than half the time in my experience it’s usually a failing sata cable or power-supply if the hard drive continues to functions on boot but eventually goes to degraded due to checksum/read errors. Sometimes the data is just fine but a read error occurs between the drive and controller which ZFS interprets as a failng disk
  - 867-5309 6y ago
    modern drives have onboard diagnostics which can report errors directly to the OS. I'd be very surprised if a modern file system was unaware of this. I'd also be surprised if users of alternative file systems didn't test a suspected failing drive before concluding it was faulty
    - redflame7 6y ago
      The error is occurring between the drive and the sata controller on the mobo. There’s also the whole issue of whether the OS or the drive is responsible for error handling. If the the drives are SAS drives, typically the OS uses fire and forget methodologies for write commands. Other drives might not support that and require zfs to handle errors at the software/kernel level
      - 867-5309 6y ago
        replacing the cable / testing the disk on an isolated system would confirm this
        blueflame7 6y ago
        It would but I encountered a situation once, where when I would remove the drives of the array and test the disk 1 by 1 they would all work. It wasn't until all the drives were powered on and under max write stress would the power-supply under-volt to drives randomly making it appear as multiple drives were all failing.
    - blibble 6y ago
      the great thing about zfs is you don't have to care, at all as long as it writes and reads back (most of the time): ZFS will deal with it
    - ianhowson 6y ago
      Hard drives lie. SMART data is not thorough and it is not trustworthy. Across dozens of drive failures over the years, I've never seen SMART predict a failure before ZFS detected data loss or the OS reported 'drive is slow'. SMART is not worth the effort to monitor. Filesystems don't read SMART data. You might have a separate daemon which monitors SMART. ZFS checksumming is amazing for this. You know, without doubt, which file(s) are bad. You can still use failing or unknown quality drives because the checksumming will protect you from silent data corruption.
      - 6y ago
- aljarry 6y ago
  One pro of ZFS over RAID1 is that ZFS is file based and not volume based, so in case of recreating mirror disks (or general resilvering) you only copy data, not the whole volume.
- throw0101a 6y ago
  > Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening. You're not wrong. And if there's a mechanical issue with a drive and it dies / stops spinning, then you'll probably get an alert and can swap it. But if there's any kind of bit rot or data corruption, and it only happens on one side of the (traditional) RAID1 mirror, how will you (a) know that it actually occurred, or (b) know which side is good bits and which side has bad bits? With ZFS and checksums, you can be confident that your data is still healthy. Depending on the data, this may be an important consideration. zfs send-recv is also very handy for doing incremental backups.
  - suncore 6y ago
    What ZFS does here is not enough for me. I need to be alerted to changes done also above the file system layer, like by malware or accidental deletion etc. Only way I have found to solve that is to have checksum tool above the file system. If something is wrong, I restore from backup, so zfs does not give me anything...
    - tsuraan 6y ago
      I don't personally do this, but if you're taking scheduled snapshots of your ZFS filesystems, you could also have a scheduled job (say, nightly) that emails you a "zfs diff" [0] between the current snapshot and the one from 24 hours ago. It won't tell you that you've been hit by malware, but an unexpected spike in changes could be something worth investigating. [0] - https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html
    - rainbowzootsuit 6y ago
      You can do a zfs diff command on two snapshots or a snapshot and current dataset as zfs takes a just-in-time snapshot of the live dataset.
    - throw0101a 6y ago
      > I need to be alerted to changes done also above the file system layer [...] As others have mentioned, would "zfs diff ..." be useful? * https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-differences-using-zfs-diff/ https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-di... As the name suggests, "snapshots" are read-only and so cannot be altered. You could either copy/rsync the modified file/s to the live location, or do a rollback to a particular snapshot: * https://www.thegeekdiary.com/solaris-zfs-how-to-create-rename-rollback-destroy-a-zfs-snapshot/ https://www.thegeekdiary.com/solaris-zfs-how-to-create-renam... If the machine is compromised in some way, you could reinstall and do a "zfs send-recv" of a pool from a remote system.
buster 6y ago
Strange. I have some WD disks in my qnap running 24/7 for 8 years now, without a single problem. I'm planning on reusing them in my new NAS..
- puzzlingcaptcha 6y ago
  I have a WD Purple I bought in 2014 and it's been running continuously since. 18 months seems extremely short.
  - simon1573 6y ago
    Isn't Purple for things like camera feeds? Good on writes, bad on reads.
    - puzzlingcaptcha 6y ago
      This seems to be the conventional wisdom but I can't say I've experienced it in practice. They have less cache than Reds and the firmware may be tuned for writes but I think it's mostly down to different market positioning/branding & price. You can compare the two: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-purple-hdd/data-sheet-wd-purple-series-2879-800012.pdf https://documents.westerndigital.com/content/dam/doc-library... and https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-hdd/product-brief-western-digital-wd-red-hdd.pdf https://documents.westerndigital.com/content/dam/doc-library...
- rewtraw 6y ago
  Same here, I have over a dozen WD Red drives with over 70,000 power on hours and all pass their SMART tests.
mhaberl 6y ago
What kind of disks and brand do you use? Are you doing some kind of disk-intensive work? I am asking because this sounds really short to me and I am wondering if I am extremely lucky never to have a failed disk in more than 25 years, mostly with more than 3 computers at a time with only consumer grade disk of random brands. I only swich them because for better performance/capacity, but some of them are almost 10 years old.
- Osiris 6y ago
  I still have a 1.6gb IDE drive that is functional. I have drives with 70k hours that still run. I've had 2 drives fail in the last 30 years. It may be rare but when it happens you'll really wish you had a backup.
- formerly_proven 6y ago
  Like my fellow commenter, I also do have a few drives with over 50k head hours on them that still work fine, but I also had probably more than half a dozen drives fail on me or break (not including Sandforce SSDs, that'd be unfair), including one that had all of my early projects on it. Really wish I had a backup of that. Now I have a backup of everything and whenever I have to think about how much storage I need for something I always take that number and multiply it by three -- if it's only stored once or twice, it really can't be more important than /tmp.
  - jsjohnst 6y ago
    Interesting for me, I’ve had two drive failures, both in the past year, but otherwise nothing for the past five+ years. Both were data center class drives (Seagate Exios) and died within the first 2 months of purchasing them. I have over 20 consumer grade drives with >50,000hrs of reported power on time, of which 3-4 are >70,000hrs. They constantly see writes, but I’m definitely not pushing max IOPs by no means on a regular basis. I’d say around 1-2% of the time they are under very heavy read/write activity, otherwise just constant appending of data at around 50Mb/sec across the entire RAID pool.
    - 6y ago
nuker 6y ago
> so that happened every 18 months or so Might be power supply issue.
3np 6y ago
Agreed. I've been aware of and thinking to check out ZFS for over a decade because all these weird nerds keep talking about it but it seemed so involved and complicated. One simple command each to: * create a zpool * create a dataset (persistently mounted across reboots, with optional encryption and compression etc) While, yes, properly tweaking ZFS for performance and making proper decisions on things like recordsizes, L2ARC and SLOG requires a bit of deeper understanding of how it works, the CLI is very approachable and the man pages are straight-forward and easy to understand for a beginner. Compared the alternatives of cryptsetup/mdadm/lvm/lvcache, it's such a breath of fresh air and a lot easier to work with. It's unified and intuitive and things just make sense. The big game-changer for me is its caching mechanisms. For workloads that already have efficient caching (mature databases, for example), it might not make a big difference, but for things doing redundant IO it can improve the performance by an order of magnitude in a way the Linux page cache just can't. Put a fast NVMe as L2ARC in front of a mirror of slow but inexpensive disks and you can eat the cake and have it. Oh, and native compression (currently lz4 but zstd is coming) that barely impacts performance (and sometimes improves it). I'm one of those weird guys now and even my nerd friends don't get me anymore :')
bsharitt 6y ago
> I was using just regular desktop grade disks so that happened every 18 months or so. That seems unusually bad. Up until earlier this month my old raid array has been mostly desktop drives(just upgraded it to IronWolf NAS drives though). I did finally do the upgrade to new drives because one died, but that drive was manufactured in 2011, and the rest of the drives are from 2010, 2013, and 2015. To be fair I've not gone five years with out a failure, I think the 2015 drive was added about 3 years ago(being my actual desktop drive before that), but otherwise have had good luck. In fact, I think my most recent failure may have been accelerated by my server being put into a temporary case with poor air flow.
cm2187 6y ago
I like to separate storage from the rest. Had various models of synology for nearly 10 years now and it’s the ideal solution for stopping to think and worry about it (combined with UPS). Running VMs, etc, I’d rather do that on a different box. You need to reboot frequently, make some experimental changes, modify the hardware, etc. Better to do that away from your data.

emilburzo 6y ago

Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening. Also, 18 months sounds kind of short, I usually get around 5 years of use from hard drives. What brand are you using? I highly recommend checking out the quarterly Backblaze HDD failure reports. Source: been running a server (read: consumer headless desktop) for years without issues.

redflame7 6y ago
More than half the time in my experience it’s usually a failing sata cable or power-supply if the hard drive continues to functions on boot but eventually goes to degraded due to checksum/read errors. Sometimes the data is just fine but a read error occurs between the drive and controller which ZFS interprets as a failng disk
- 867-5309 6y ago
  modern drives have onboard diagnostics which can report errors directly to the OS. I'd be very surprised if a modern file system was unaware of this. I'd also be surprised if users of alternative file systems didn't test a suspected failing drive before concluding it was faulty
  - redflame7 6y ago
    The error is occurring between the drive and the sata controller on the mobo. There’s also the whole issue of whether the OS or the drive is responsible for error handling. If the the drives are SAS drives, typically the OS uses fire and forget methodologies for write commands. Other drives might not support that and require zfs to handle errors at the software/kernel level
    - 867-5309 6y ago
      replacing the cable / testing the disk on an isolated system would confirm this
      - blueflame7 6y ago
        It would but I encountered a situation once, where when I would remove the drives of the array and test the disk 1 by 1 they would all work. It wasn't until all the drives were powered on and under max write stress would the power-supply under-volt to drives randomly making it appear as multiple drives were all failing.
  - blibble 6y ago
    the great thing about zfs is you don't have to care, at all as long as it writes and reads back (most of the time): ZFS will deal with it
  - ianhowson 6y ago
    Hard drives lie. SMART data is not thorough and it is not trustworthy. Across dozens of drive failures over the years, I've never seen SMART predict a failure before ZFS detected data loss or the OS reported 'drive is slow'. SMART is not worth the effort to monitor. Filesystems don't read SMART data. You might have a separate daemon which monitors SMART. ZFS checksumming is amazing for this. You know, without doubt, which file(s) are bad. You can still use failing or unknown quality drives because the checksumming will protect you from silent data corruption.
    - 6y ago
aljarry 6y ago
One pro of ZFS over RAID1 is that ZFS is file based and not volume based, so in case of recreating mirror disks (or general resilvering) you only copy data, not the whole volume.
throw0101a 6y ago
> Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening. You're not wrong. And if there's a mechanical issue with a drive and it dies / stops spinning, then you'll probably get an alert and can swap it. But if there's any kind of bit rot or data corruption, and it only happens on one side of the (traditional) RAID1 mirror, how will you (a) know that it actually occurred, or (b) know which side is good bits and which side has bad bits? With ZFS and checksums, you can be confident that your data is still healthy. Depending on the data, this may be an important consideration. zfs send-recv is also very handy for doing incremental backups.
- suncore 6y ago
  What ZFS does here is not enough for me. I need to be alerted to changes done also above the file system layer, like by malware or accidental deletion etc. Only way I have found to solve that is to have checksum tool above the file system. If something is wrong, I restore from backup, so zfs does not give me anything...
  - tsuraan 6y ago
    I don't personally do this, but if you're taking scheduled snapshots of your ZFS filesystems, you could also have a scheduled job (say, nightly) that emails you a "zfs diff" [0] between the current snapshot and the one from 24 hours ago. It won't tell you that you've been hit by malware, but an unexpected spike in changes could be something worth investigating. [0] - https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html
  - rainbowzootsuit 6y ago
    You can do a zfs diff command on two snapshots or a snapshot and current dataset as zfs takes a just-in-time snapshot of the live dataset.
  - throw0101a 6y ago
    > I need to be alerted to changes done also above the file system layer [...] As others have mentioned, would "zfs diff ..." be useful? * https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-differences-using-zfs-diff/ https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-di... As the name suggests, "snapshots" are read-only and so cannot be altered. You could either copy/rsync the modified file/s to the live location, or do a rollback to a particular snapshot: * https://www.thegeekdiary.com/solaris-zfs-how-to-create-rename-rollback-destroy-a-zfs-snapshot/ https://www.thegeekdiary.com/solaris-zfs-how-to-create-renam... If the machine is compromised in some way, you could reinstall and do a "zfs send-recv" of a pool from a remote system.

buster 6y ago

Strange. I have some WD disks in my qnap running 24/7 for 8 years now, without a single problem. I'm planning on reusing them in my new NAS..

puzzlingcaptcha 6y ago
I have a WD Purple I bought in 2014 and it's been running continuously since. 18 months seems extremely short.
- simon1573 6y ago
  Isn't Purple for things like camera feeds? Good on writes, bad on reads.
  - puzzlingcaptcha 6y ago
    This seems to be the conventional wisdom but I can't say I've experienced it in practice. They have less cache than Reds and the firmware may be tuned for writes but I think it's mostly down to different market positioning/branding & price. You can compare the two: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-purple-hdd/data-sheet-wd-purple-series-2879-800012.pdf https://documents.westerndigital.com/content/dam/doc-library... and https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-hdd/product-brief-western-digital-wd-red-hdd.pdf https://documents.westerndigital.com/content/dam/doc-library...
rewtraw 6y ago
Same here, I have over a dozen WD Red drives with over 70,000 power on hours and all pass their SMART tests.

mhaberl 6y ago

What kind of disks and brand do you use? Are you doing some kind of disk-intensive work? I am asking because this sounds really short to me and I am wondering if I am extremely lucky never to have a failed disk in more than 25 years, mostly with more than 3 computers at a time with only consumer grade disk of random brands. I only swich them because for better performance/capacity, but some of them are almost 10 years old.

Osiris 6y ago
I still have a 1.6gb IDE drive that is functional. I have drives with 70k hours that still run. I've had 2 drives fail in the last 30 years. It may be rare but when it happens you'll really wish you had a backup.
formerly_proven 6y ago
Like my fellow commenter, I also do have a few drives with over 50k head hours on them that still work fine, but I also had probably more than half a dozen drives fail on me or break (not including Sandforce SSDs, that'd be unfair), including one that had all of my early projects on it. Really wish I had a backup of that. Now I have a backup of everything and whenever I have to think about how much storage I need for something I always take that number and multiply it by three -- if it's only stored once or twice, it really can't be more important than /tmp.
- jsjohnst 6y ago
  Interesting for me, I’ve had two drive failures, both in the past year, but otherwise nothing for the past five+ years. Both were data center class drives (Seagate Exios) and died within the first 2 months of purchasing them. I have over 20 consumer grade drives with >50,000hrs of reported power on time, of which 3-4 are >70,000hrs. They constantly see writes, but I’m definitely not pushing max IOPs by no means on a regular basis. I’d say around 1-2% of the time they are under very heavy read/write activity, otherwise just constant appending of data at around 50Mb/sec across the entire RAID pool.
  - 6y ago

nuker 6y ago

> so that happened every 18 months or so Might be power supply issue.

3np 6y ago

Agreed. I've been aware of and thinking to check out ZFS for over a decade because all these weird nerds keep talking about it but it seemed so involved and complicated. One simple command each to: * create a zpool * create a dataset (persistently mounted across reboots, with optional encryption and compression etc) While, yes, properly tweaking ZFS for performance and making proper decisions on things like recordsizes, L2ARC and SLOG requires a bit of deeper understanding of how it works, the CLI is very approachable and the man pages are straight-forward and easy to understand for a beginner. Compared the alternatives of cryptsetup/mdadm/lvm/lvcache, it's such a breath of fresh air and a lot easier to work with. It's unified and intuitive and things just make sense. The big game-changer for me is its caching mechanisms. For workloads that already have efficient caching (mature databases, for example), it might not make a big difference, but for things doing redundant IO it can improve the performance by an order of magnitude in a way the Linux page cache just can't. Put a fast NVMe as L2ARC in front of a mirror of slow but inexpensive disks and you can eat the cake and have it. Oh, and native compression (currently lz4 but zstd is coming) that barely impacts performance (and sometimes improves it). I'm one of those weird guys now and even my nerd friends don't get me anymore :')

bsharitt 6y ago

> I was using just regular desktop grade disks so that happened every 18 months or so. That seems unusually bad. Up until earlier this month my old raid array has been mostly desktop drives(just upgraded it to IronWolf NAS drives though). I did finally do the upgrade to new drives because one died, but that drive was manufactured in 2011, and the rest of the drives are from 2010, 2013, and 2015. To be fair I've not gone five years with out a failure, I think the 2015 drive was added about 3 years ago(being my actual desktop drive before that), but otherwise have had good luck. In fact, I think my most recent failure may have been accelerated by my server being put into a temporary case with poor air flow.

cm2187 6y ago

I like to separate storage from the rest. Had various models of synology for nearly 10 years now and it’s the ideal solution for stopping to think and worry about it (combined with UPS). Running VMs, etc, I’d rather do that on a different box. You need to reboot frequently, make some experimental changes, modify the hardware, etc. Better to do that away from your data.