4 ms·

I’m not an expert whatsoever but what I’ve been doing for my NAS is using mirrored VDEVs. Started with one and later on added a couple more drives for a second

• edmundsauto 3y ago
  I’ve read this is suboptimal because you are now stressing the drive that has the only copy of your data to rebuild. what are your thoughts?
  • deadbunny 3y ago
    Mirrored vdevs resilver a lot faster than zX vdevs. Much less chance of the remaining drive dying during a resilver if it takes hours rather than days.
    • edmundsauto 3y ago
      Is the amount of data read/written the same? I'm not clear why wall time is the relevant metric, unless that's the critical driver of failure likelihood. I would have guessed it's not time but bytes.
      • keeperofdakeys 3y ago
        A mirror resilver is a relatively linear and sequential rewrite, so its very fast. Raidz resilvers require lots of random reads and writes across all the drives, and requires waiting for all drives to read data before it can be written to the replaced drive - "herding cats" sounds appropriate here.
        • didntcheck 3y ago
          That makes sense. Usually the received wisdom I hear is "RAIDZ is slower due to the parity calculations", which has always seemed dubious to me on modern CPUs, given we can do much more complex compression and encryption at way faster then disk speed (for HDDs, at least)
          • someplaceguy 3y ago
            Usually what makes RAID-Z slower is not really the parity calculations, it's the fact that each RAID-Z vdev only has the read IOPS of the slowest disk in it. So for example, a pool with 2 RAID-Z vdevs each containing 5 disks effectively has only 2x the IOPS of a single disk, while a pool with 5 mirror vdevs (i.e. RAID-10) has 10x the IOPS of a single disk. It's not a big problem if you mostly do sequential I/O but it's a huge difference if/when you do small random reads (e.g. traversing a non-cached directory tree). In the context of resilvers, currently RAID-Z pools can only be done by traversing the block tree which, due to fragmentation of CoW filesystems, usually leads to a lot of random reads/writes, while a RAID-10 pool can basically resilver a pool while doing almost fully sequential I/O which can be much, much faster.
        • edmundsauto 3y ago
          Do random reads and writes out more stress on the drive mechanism? I’m wondering if this increase in resliver time adds risk of data loss, which is probably the more important factor than clock time. (But I’m open to learning if I am wrong)
  • ilyt 3y ago
    ....vs stressing all of them for parity rebuild.
    • edmundsauto 3y ago
      Wouldn't it be better to apply stress over more drives to minimize the chances that you lose a second drive during a rebuild?
      • ErneX 3y ago
        I believe this is the article I read when I started: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...
        • ilyt 3y ago
          That talks more about RAID5/Z1 vs mirroring. You still need 6/Z2 if you want to have reasonable fault tolerance, unless you want to waste ungodly amount of space.
          • ErneX 3y ago
            My reasoning for using mirrored vdevs was more about the ease of expanding the storage than the redundancy.
            • ilyt 3y ago
              We just used RAID6... expanding by more drives is easy while keeping data waste low and the "we want to put bigger drives now" case isn't all that problematic considering that what you save you waste on low efficiency of RAID1 setup. We did had a big case so we just ran 2xRAID6 setup and that one time where it was needed we just replaced to bigger drives one by one, while using the removed ones as spares for other machines. But that's benefit of scale bigger than "a NAS server".
  • ErneX 3y ago
    I am backing up the whole NAS in case such a failure happens, and as was mentioned in other replies getting the replacement drive in the pool is way quicker on a mirrored vdev.