4 ms·

As always with RAID-style setups, there’s an inevitable trade off of cost vs capacity vs performance. There’s still a place for RAIDZ/RAIDZ2, and in my opinion

As always with RAID-style setups, there’s an inevitable trade off of cost vs capacity vs performance. There’s still a place for RAIDZ/RAIDZ2, and in my opinion that place is storing bulk data that isn’t too heavily accessed or that needs to be stored with an eye towards keeping £/GB down. Yes, mirrors are faster. Yes, mirrors are easier to expand. But across 12 4TB disks that is 24TB instead of 40TB with RAIDZ2 - and that’s a lot of capacity to lose if you’re on a budget. The rebuild times in this post seem high to me, though. I replaced 7x 2TB disks (nearly full) in a raidz1 in a backup pool with larger drives in about 30 hours.

minimaul 6y ago
The one thing in the article I agree with without reservation though is that you should always have a backup of your pool! ZFS makes doing good backups easy, with zfs send | zfs receive.
- gh02t 6y ago
  I mean, is this controversial? The problem for a budget conscious home user is: where do you back up to? Doing a complete backup to a remote storage provider isn't a practical option for many users, which leaves the affordable options being to only back up critical data and/or use something like a big USB external drive. Neither of these options are great, so in reality many people end up relying on RAID or mirroring to provide some measure of redundancy regardless of whether or not it's a good idea.
  - benlivengood 6y ago
    I just started playing around with s3blkdevd. S3-compatible storage at backblaze B2 is $0.005/GB-month and I'll see if I can manage a vdev made out of a huge nbd disk. Topical to the article, backblaze has piles of RAID6 and appears to be happy with it. Another thought has been to find a ZFS-friend in another part of the world and exchange snapshots. Inline encryption makes that a little more viable now. Right now I'm sending incremental snapshots to AWS Glacier and have another local box mirroring the data (mostly to fully verify the snapshots before sending them to Glacier).
  - unethical_ban 6y ago
    I have a raidz2 that I keep everything on: (A) Personal, (B) core media, and (C) all other media. I backup A and B on a workstation drive, and I further backup A to S3 Glacier for $1/month/100GB, syncing every few months. It isn't all or nothing, 3-copies vs. striped pool. Mirrors are fine if you have quick access to fix dead drives, but I like the idea of being able to lose any two drives vs. only one drive in a given mirror. And while I currently have a 4 drive RAIDz2 (same disk utilization as a mirror) I wish I had a 4+2 array, which I'll probably build next year.
    - gh02t 6y ago
      Yeah, this is basically what I do as well, I didn't mean to imply it is all or nothing. The practical solution is tiered backup of the stuff you really care about + some local fault tolerance for stuff you don't care about as much. What I wanted to point out is that of course backing up everything is the solution per the parent comment, but the reality is that costs for punting and mindlessly backing up say 10 TB of data properly are still fairly high. So you pick and choose the important stuff and do the best you can for the rest.
magicalhippo 6y ago
> The rebuild times in this post seem high to me, though. Two factors. Disks are getting large, and the rebuild time for RAID-Z[1] is dependent on the fragmentation. In combination it means it can take ages. I just had to replace a failing WD Red 3TB[2] in an old 4xRAID-Z1 pool, and it took 9 hours. That was a single 3TB disk. The disks in my new pool are 14TB and 16TB. [1]: https://youtu.be/Efl0Kv_hXwY https://youtu.be/Efl0Kv_hXwY [2]: power-on hours in SMART showed over 7 years
- throw0101a 6y ago
  OpenZFS 2.0 has sequential rebuilds: > The sequential reconstruction feature adds a more traditional RAID rebuild mechanism to ZFS. Specifically, it allows for mirror vdevs to be rebuilt in LBA order. Depending on the pools average block size, overall fragmentation, and the performance characteristics of the devices (SMR) sequential reconstruction can restore redundancy in less time than a traditional healing resilver. However, it cannot verify block checksums as part of the rebuild. Therefore a scrub is automatically started when the last active sequential resilver completes. * https://github.com/openzfs/zfs/pull/10349 https://github.com/openzfs/zfs/pull/10349
  - magicalhippo 6y ago
    Yeah I linked to the OpenZFS Developer Summit video on the feature, but forgot to mention it in my post.
zepearl 6y ago
Btw., I understand that with OpenZFS 2 and its new "sequential resilvering" feature ( https://arstechnica.com/gadgets/2020/12/openzfs-2-0-release-unifies-linux-bsd-and-adds-tons-of-new-features/ https://arstechnica.com/gadgets/2020/12/openzfs-2-0-release-... ) rebuilding should be faster or should at least put less pressure on the drives (less random I/O)... . Edit: sorry, just noticed that this was already mentioned in another subthread.
- bewaretheirs 6y ago
  There's also the new "special" vdev feature (which allow you to segregate metadata onto separate ssd mirrors) which should further speed up the remaining unavoidable random reads during scrub/resilver (for metadata traversal).

minimaul 6y ago

The one thing in the article I agree with without reservation though is that you should always have a backup of your pool! ZFS makes doing good backups easy, with zfs send | zfs receive.

gh02t 6y ago
I mean, is this controversial? The problem for a budget conscious home user is: where do you back up to? Doing a complete backup to a remote storage provider isn't a practical option for many users, which leaves the affordable options being to only back up critical data and/or use something like a big USB external drive. Neither of these options are great, so in reality many people end up relying on RAID or mirroring to provide some measure of redundancy regardless of whether or not it's a good idea.
- benlivengood 6y ago
  I just started playing around with s3blkdevd. S3-compatible storage at backblaze B2 is $0.005/GB-month and I'll see if I can manage a vdev made out of a huge nbd disk. Topical to the article, backblaze has piles of RAID6 and appears to be happy with it. Another thought has been to find a ZFS-friend in another part of the world and exchange snapshots. Inline encryption makes that a little more viable now. Right now I'm sending incremental snapshots to AWS Glacier and have another local box mirroring the data (mostly to fully verify the snapshots before sending them to Glacier).
- unethical_ban 6y ago
  I have a raidz2 that I keep everything on: (A) Personal, (B) core media, and (C) all other media. I backup A and B on a workstation drive, and I further backup A to S3 Glacier for $1/month/100GB, syncing every few months. It isn't all or nothing, 3-copies vs. striped pool. Mirrors are fine if you have quick access to fix dead drives, but I like the idea of being able to lose any two drives vs. only one drive in a given mirror. And while I currently have a 4 drive RAIDz2 (same disk utilization as a mirror) I wish I had a 4+2 array, which I'll probably build next year.
  - gh02t 6y ago
    Yeah, this is basically what I do as well, I didn't mean to imply it is all or nothing. The practical solution is tiered backup of the stuff you really care about + some local fault tolerance for stuff you don't care about as much. What I wanted to point out is that of course backing up everything is the solution per the parent comment, but the reality is that costs for punting and mindlessly backing up say 10 TB of data properly are still fairly high. So you pick and choose the important stuff and do the best you can for the rest.

magicalhippo 6y ago

> The rebuild times in this post seem high to me, though. Two factors. Disks are getting large, and the rebuild time for RAID-Z[1] is dependent on the fragmentation. In combination it means it can take ages. I just had to replace a failing WD Red 3TB[2] in an old 4xRAID-Z1 pool, and it took 9 hours. That was a single 3TB disk. The disks in my new pool are 14TB and 16TB. [1]: https://youtu.be/Efl0Kv_hXwY https://youtu.be/Efl0Kv_hXwY [2]: power-on hours in SMART showed over 7 years

throw0101a 6y ago
OpenZFS 2.0 has sequential rebuilds: > The sequential reconstruction feature adds a more traditional RAID rebuild mechanism to ZFS. Specifically, it allows for mirror vdevs to be rebuilt in LBA order. Depending on the pools average block size, overall fragmentation, and the performance characteristics of the devices (SMR) sequential reconstruction can restore redundancy in less time than a traditional healing resilver. However, it cannot verify block checksums as part of the rebuild. Therefore a scrub is automatically started when the last active sequential resilver completes. * https://github.com/openzfs/zfs/pull/10349 https://github.com/openzfs/zfs/pull/10349
- magicalhippo 6y ago
  Yeah I linked to the OpenZFS Developer Summit video on the feature, but forgot to mention it in my post.

zepearl 6y ago

Btw., I understand that with OpenZFS 2 and its new "sequential resilvering" feature ( https://arstechnica.com/gadgets/2020/12/openzfs-2-0-release-unifies-linux-bsd-and-adds-tons-of-new-features/ https://arstechnica.com/gadgets/2020/12/openzfs-2-0-release-... ) rebuilding should be faster or should at least put less pressure on the drives (less random I/O)... . Edit: sorry, just noticed that this was already mentioned in another subthread.

bewaretheirs 6y ago
There's also the new "special" vdev feature (which allow you to segregate metadata onto separate ssd mirrors) which should further speed up the remaining unavoidable random reads during scrub/resilver (for metadata traversal).