4 ms·

Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening. Also, 18 months sounds k

Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening. Also, 18 months sounds kind of short, I usually get around 5 years of use from hard drives. What brand are you using? I highly recommend checking out the quarterly Backblaze HDD failure reports. Source: been running a server (read: consumer headless desktop) for years without issues.

redflame7 6y ago
More than half the time in my experience it’s usually a failing sata cable or power-supply if the hard drive continues to functions on boot but eventually goes to degraded due to checksum/read errors. Sometimes the data is just fine but a read error occurs between the drive and controller which ZFS interprets as a failng disk
- 867-5309 6y ago
  modern drives have onboard diagnostics which can report errors directly to the OS. I'd be very surprised if a modern file system was unaware of this. I'd also be surprised if users of alternative file systems didn't test a suspected failing drive before concluding it was faulty
  - redflame7 6y ago
    The error is occurring between the drive and the sata controller on the mobo. There’s also the whole issue of whether the OS or the drive is responsible for error handling. If the the drives are SAS drives, typically the OS uses fire and forget methodologies for write commands. Other drives might not support that and require zfs to handle errors at the software/kernel level
    - 867-5309 6y ago
      replacing the cable / testing the disk on an isolated system would confirm this
      - blueflame7 6y ago
        It would but I encountered a situation once, where when I would remove the drives of the array and test the disk 1 by 1 they would all work. It wasn't until all the drives were powered on and under max write stress would the power-supply under-volt to drives randomly making it appear as multiple drives were all failing.
  - blibble 6y ago
    the great thing about zfs is you don't have to care, at all as long as it writes and reads back (most of the time): ZFS will deal with it
  - ianhowson 6y ago
    Hard drives lie. SMART data is not thorough and it is not trustworthy. Across dozens of drive failures over the years, I've never seen SMART predict a failure before ZFS detected data loss or the OS reported 'drive is slow'. SMART is not worth the effort to monitor. Filesystems don't read SMART data. You might have a separate daemon which monitors SMART. ZFS checksumming is amazing for this. You know, without doubt, which file(s) are bad. You can still use failing or unknown quality drives because the checksumming will protect you from silent data corruption.
    - toast0 6y ago
      SMART marking drives as failed is usually super late. But if you monitor the raw values for the important parameters (unreadable, reallocated, etc sectors), I've had good luck with replacing drives before software notices. At least for spinning drives; SSD failues were way more rare, but resulted in the drives completely disappearing from the bus, and no reliable prefail indicators. I did have one SSD go through a big reallocation that tanked throughput and the alerts from throughput and the alerts from SMART thresholds fired simultaneously. Of course, that's great for a server farm; in home use monitoring is a lot less structured.
    - pixl97 6y ago
      Expensive enterprise drives are generally better at smart stats. Of course your mileage varies by vendor. Desktop drives on the other hand are completely untrustworthy.
aljarry 6y ago
One pro of ZFS over RAID1 is that ZFS is file based and not volume based, so in case of recreating mirror disks (or general resilvering) you only copy data, not the whole volume.
throw0101a 6y ago
> Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening. You're not wrong. And if there's a mechanical issue with a drive and it dies / stops spinning, then you'll probably get an alert and can swap it. But if there's any kind of bit rot or data corruption, and it only happens on one side of the (traditional) RAID1 mirror, how will you (a) know that it actually occurred, or (b) know which side is good bits and which side has bad bits? With ZFS and checksums, you can be confident that your data is still healthy. Depending on the data, this may be an important consideration. zfs send-recv is also very handy for doing incremental backups.
- suncore 6y ago
  What ZFS does here is not enough for me. I need to be alerted to changes done also above the file system layer, like by malware or accidental deletion etc. Only way I have found to solve that is to have checksum tool above the file system. If something is wrong, I restore from backup, so zfs does not give me anything...
  - tsuraan 6y ago
    I don't personally do this, but if you're taking scheduled snapshots of your ZFS filesystems, you could also have a scheduled job (say, nightly) that emails you a "zfs diff" [0] between the current snapshot and the one from 24 hours ago. It won't tell you that you've been hit by malware, but an unexpected spike in changes could be something worth investigating. [0] - https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html
  - rainbowzootsuit 6y ago
    You can do a zfs diff command on two snapshots or a snapshot and current dataset as zfs takes a just-in-time snapshot of the live dataset.
  - throw0101a 6y ago
    > I need to be alerted to changes done also above the file system layer [...] As others have mentioned, would "zfs diff ..." be useful? * https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-differences-using-zfs-diff/ https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-di... As the name suggests, "snapshots" are read-only and so cannot be altered. You could either copy/rsync the modified file/s to the live location, or do a rollback to a particular snapshot: * https://www.thegeekdiary.com/solaris-zfs-how-to-create-rename-rollback-destroy-a-zfs-snapshot/ https://www.thegeekdiary.com/solaris-zfs-how-to-create-renam... If the machine is compromised in some way, you could reinstall and do a "zfs send-recv" of a pool from a remote system.

redflame7 6y ago

More than half the time in my experience it’s usually a failing sata cable or power-supply if the hard drive continues to functions on boot but eventually goes to degraded due to checksum/read errors. Sometimes the data is just fine but a read error occurs between the drive and controller which ZFS interprets as a failng disk

867-5309 6y ago
modern drives have onboard diagnostics which can report errors directly to the OS. I'd be very surprised if a modern file system was unaware of this. I'd also be surprised if users of alternative file systems didn't test a suspected failing drive before concluding it was faulty
- redflame7 6y ago
  The error is occurring between the drive and the sata controller on the mobo. There’s also the whole issue of whether the OS or the drive is responsible for error handling. If the the drives are SAS drives, typically the OS uses fire and forget methodologies for write commands. Other drives might not support that and require zfs to handle errors at the software/kernel level
  - 867-5309 6y ago
    replacing the cable / testing the disk on an isolated system would confirm this
    - blueflame7 6y ago
      It would but I encountered a situation once, where when I would remove the drives of the array and test the disk 1 by 1 they would all work. It wasn't until all the drives were powered on and under max write stress would the power-supply under-volt to drives randomly making it appear as multiple drives were all failing.
- blibble 6y ago
  the great thing about zfs is you don't have to care, at all as long as it writes and reads back (most of the time): ZFS will deal with it
- ianhowson 6y ago
  Hard drives lie. SMART data is not thorough and it is not trustworthy. Across dozens of drive failures over the years, I've never seen SMART predict a failure before ZFS detected data loss or the OS reported 'drive is slow'. SMART is not worth the effort to monitor. Filesystems don't read SMART data. You might have a separate daemon which monitors SMART. ZFS checksumming is amazing for this. You know, without doubt, which file(s) are bad. You can still use failing or unknown quality drives because the checksumming will protect you from silent data corruption.
  - toast0 6y ago
    SMART marking drives as failed is usually super late. But if you monitor the raw values for the important parameters (unreadable, reallocated, etc sectors), I've had good luck with replacing drives before software notices. At least for spinning drives; SSD failues were way more rare, but resulted in the drives completely disappearing from the bus, and no reliable prefail indicators. I did have one SSD go through a big reallocation that tanked throughput and the alerts from throughput and the alerts from SMART thresholds fired simultaneously. Of course, that's great for a server farm; in home use monitoring is a lot less structured.
  - pixl97 6y ago
    Expensive enterprise drives are generally better at smart stats. Of course your mileage varies by vendor. Desktop drives on the other hand are completely untrustworthy.

aljarry 6y ago

One pro of ZFS over RAID1 is that ZFS is file based and not volume based, so in case of recreating mirror disks (or general resilvering) you only copy data, not the whole volume.

throw0101a 6y ago

> Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening. You're not wrong. And if there's a mechanical issue with a drive and it dies / stops spinning, then you'll probably get an alert and can swap it. But if there's any kind of bit rot or data corruption, and it only happens on one side of the (traditional) RAID1 mirror, how will you (a) know that it actually occurred, or (b) know which side is good bits and which side has bad bits? With ZFS and checksums, you can be confident that your data is still healthy. Depending on the data, this may be an important consideration. zfs send-recv is also very handy for doing incremental backups.

suncore 6y ago
What ZFS does here is not enough for me. I need to be alerted to changes done also above the file system layer, like by malware or accidental deletion etc. Only way I have found to solve that is to have checksum tool above the file system. If something is wrong, I restore from backup, so zfs does not give me anything...
- tsuraan 6y ago
  I don't personally do this, but if you're taking scheduled snapshots of your ZFS filesystems, you could also have a scheduled job (say, nightly) that emails you a "zfs diff" [0] between the current snapshot and the one from 24 hours ago. It won't tell you that you've been hit by malware, but an unexpected spike in changes could be something worth investigating. [0] - https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html
- rainbowzootsuit 6y ago
  You can do a zfs diff command on two snapshots or a snapshot and current dataset as zfs takes a just-in-time snapshot of the live dataset.
- throw0101a 6y ago
  > I need to be alerted to changes done also above the file system layer [...] As others have mentioned, would "zfs diff ..." be useful? * https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-differences-using-zfs-diff/ https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-di... As the name suggests, "snapshots" are read-only and so cannot be altered. You could either copy/rsync the modified file/s to the live location, or do a rollback to a particular snapshot: * https://www.thegeekdiary.com/solaris-zfs-how-to-create-rename-rollback-destroy-a-zfs-snapshot/ https://www.thegeekdiary.com/solaris-zfs-how-to-create-renam... If the machine is compromised in some way, you could reinstall and do a "zfs send-recv" of a pool from a remote system.