5 ms·

That would be awfully hard to do as I am not using Linux (: 1. Yes, two mirrored vdevs are not the highest performance configuration for four devices, but they

That would be awfully hard to do as I am not using Linux (: 1. Yes, two mirrored vdevs are not the highest performance configuration for four devices, but they beat a single raidz vdev, which was the suggestion from the prior poster. 2. Random reads are more important than sequential for desktop responsiveness, and mirrors can serve different reads from each disk in the mirror, effectively doubling IOPS (minus overheads). 5. I am not sure I understand here. It seems like you're suggesting 3 data drives and a parity drive. This would allow at most one drive loss before data loss. Two mirrors allow at least one drive loss and up to two drive losses. If there were two parity drives, this would allow up to two drive losses without data losses, but it seems this would have to reduce your claim in point (1) to being 2x performance, since two drives would have to be used for parity writes.

justsomehnguy 3y ago
Ouch. 2. Double IOPS sounds good on paper, at 100% it is still to slow, 100 IOPS at best. 5. Wait. Mirrored mirror (RAID11) on 4 drives would net you only 10TB usable. 'Mirrored pair' sounds like an ordinary RAID10 which can tolerate 1 drive loss because it's no longer redundant for half of the data. 'It can tolerate another loss on the still redundant data' is... like pulling out - works most of the time, but some become parents on the first try. Hence RAID5 have the same redundancy tolerance (1 drive loss for losing R.) then, anything other is gambling. The better question - do you have a proper backup to protect from a logical corruption? Even RAID11 wouldn't help from rf -rf in the wrong path.
- greggyb 3y ago
  It is objectively measurable and subjectively it feels better than tests on a single drive for day to day activity. Objective and subjective improvement is good enough for me. Two mirrors == two vdevs, each a mirror of two disks: $ zpool status pool: zroot state: ONLINE scan: scrub repaired 0B in 02:01:39 with 0 errors on Wed Aug 16 14:08:39 2023 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada2p1.eli ONLINE 0 0 0 ada3p1.eli ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ada0p1.eli ONLINE 0 0 0 ada1p1.eli ONLINE 0 0 0 In a best case failure scenario, I can lose one disk from each mirror. In a worst case failure scenario, I can lose one disk. With one parity disk out of four drives (for 30TB usable), in a best case failure scenario, I can lose one disk. In a worst case failure scenario, I can lose one. These two descriptions of resiliency simply are not the same. I would obviously not run the pool in a degraded state for any longer than it takes to resilver the degraded mirror. Assuming disk failures are independent and one disk has already failed: - 2 mirror vdevs: 1/3 chance that a second disk failure causes data loss, 2/3 chance that it affects the other mirror - 4 disks w/ 1 parity: 100% chance that a second disk failure causes data loss. Again, this simply is not the same characterization of resilience. Please do not mistake this for me claiming that running this pool with three disks is anything other than a situation to remediate immediately. I take regular snapshots, and scrub regularly to ensure data integrity. Critical data is stored in at least three places, one of which is not my home.
  - srhtftw 3y ago
    > Please do not mistake this for me claiming that running this pool with three disks is anything other than a situation to remediate immediately. About a decade and a half ago I setup a 48 drive Sun 4540 this way. The OS needed 2 dedicated drives on the first controller so I assigned two neighboring drives on that controller near the intake fans as hot spares. The remaining drives were organized into mirror pairs with each drive on a separate controller. The final root was a concatenation of the pairs. This setup proved to be very robust. Although I left the company when financial troubles pushed the founders into a messy divorce, one of the founders asked me to come back a few years later after they restructured. The 4540 had been moved to a new colo but otherwise ignored because none of the admins knew Solaris or bothered to check the monitoring systems. When I returned I discovered in the time I was away 4 drives had failed but ZFS/smf happily took two of the failed drives offline and replaced them with the spares. The two other failed drives left their respective pools degraded but functional. Replacing the failed drives was easy once they arrived. Although neglected for years, none of the applications or databases that relied on that server ever suffered any downtime from the drive failures.
  - justsomehnguy 3y ago
    > These two descriptions of resiliency simply are not the same. I would obviously not run the pool in a degraded state for any longer than it takes to resilver the degraded mirror. Then you are just wasting 10TBs. I'm firmly in Murphy's camp regarding the storage troubles (hell, I can't find a specific pic of my passed dog which I know I had from 4 (four!) copies of my photos, ffs) and I treat RAID just as a mean to increase availability. In your case, if you really wouldn't run the degraded array until it is resilvered then your availability is excessive in the all cases except: how long it would take you to recover/restore your data in a catastrophic failure? And why do you have only the critical data 3-2-1ed?
    - greggyb 3y ago
      For the third time, my plan for growing the pool is based on being able to swap in new drives to trade them out with one existing drive in current mirror vdevs. As described previously, this allows me to grow two drives at a time and reduce risk of simultaneous failure in a vdev. As I mentioned before, this sort of swapping is not possible without degrading a raidz vdev. As you have brought up, this configuration optimizes resilvering speed, aka minimizing time spent in a degraded state. Resilvering a mirror is faster than resilvering a raidz, as it requires only sequential read from the good drive and sequential write to the new drive. Resilvering a raidz thrashes all drives in the vdev and requires re-calculating parity along the way. Is it more likely that one of three drives in the degraded raidz1 fails during resilvering or that the one drive in the degraded mirror fails? That is a tough question to answer. What if all the drives are the same age? Well then the likelihood is greater that a resilver will cause a cascading failure. Hence my thrice-repeated desire to use the ability to swap new drives into old vdevs to mix ages, and my mixed-brand vdevs, as described in the root of this thread. In short, resilvering is the highest-risk phase in a pool's lifecycle. I am paying an extra drive's worth of storage to optimize around this risk, and also to realize performance benefits we have discussed upthread. Again, I will state that objectively measurable improvements combined with subjective perception of better desktop performance is sufficient for me to claim that the performance improvement matters. My non-critical data is largely recoverable. Things like VM images and installers. These can be re-created and deployed from code that is in the critical portion, or downloaded from the internet. Indeed, my DR plan does not account for broad internet outages concurrent with a drive failure. A lot of the rest of the non-critical data is media that could be re-ripped from physical media (DVDs, Blu-rays). Being recoverable, it is not worth it to me to pay for cloud storage for the rest. Even "wasting" a drive's worth of storage, it is still a lot cheaper per gigabyte-year to ... you know, not pay for that. So, why do you care so much about my specific storage setup? Does it pass muster yet, or am I still someone who is wrong on the internet to you?

justsomehnguy 3y ago

Ouch. 2. Double IOPS sounds good on paper, at 100% it is still to slow, 100 IOPS at best. 5. Wait. Mirrored mirror (RAID11) on 4 drives would net you only 10TB usable. 'Mirrored pair' sounds like an ordinary RAID10 which can tolerate 1 drive loss because it's no longer redundant for half of the data. 'It can tolerate another loss on the still redundant data' is... like pulling out - works most of the time, but some become parents on the first try. Hence RAID5 have the same redundancy tolerance (1 drive loss for losing R.) then, anything other is gambling. The better question - do you have a proper backup to protect from a logical corruption? Even RAID11 wouldn't help from rf -rf in the wrong path.

greggyb 3y ago
It is objectively measurable and subjectively it feels better than tests on a single drive for day to day activity. Objective and subjective improvement is good enough for me. Two mirrors == two vdevs, each a mirror of two disks: $ zpool status pool: zroot state: ONLINE scan: scrub repaired 0B in 02:01:39 with 0 errors on Wed Aug 16 14:08:39 2023 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada2p1.eli ONLINE 0 0 0 ada3p1.eli ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ada0p1.eli ONLINE 0 0 0 ada1p1.eli ONLINE 0 0 0 In a best case failure scenario, I can lose one disk from each mirror. In a worst case failure scenario, I can lose one disk. With one parity disk out of four drives (for 30TB usable), in a best case failure scenario, I can lose one disk. In a worst case failure scenario, I can lose one. These two descriptions of resiliency simply are not the same. I would obviously not run the pool in a degraded state for any longer than it takes to resilver the degraded mirror. Assuming disk failures are independent and one disk has already failed: - 2 mirror vdevs: 1/3 chance that a second disk failure causes data loss, 2/3 chance that it affects the other mirror - 4 disks w/ 1 parity: 100% chance that a second disk failure causes data loss. Again, this simply is not the same characterization of resilience. Please do not mistake this for me claiming that running this pool with three disks is anything other than a situation to remediate immediately. I take regular snapshots, and scrub regularly to ensure data integrity. Critical data is stored in at least three places, one of which is not my home.
- srhtftw 3y ago
  > Please do not mistake this for me claiming that running this pool with three disks is anything other than a situation to remediate immediately. About a decade and a half ago I setup a 48 drive Sun 4540 this way. The OS needed 2 dedicated drives on the first controller so I assigned two neighboring drives on that controller near the intake fans as hot spares. The remaining drives were organized into mirror pairs with each drive on a separate controller. The final root was a concatenation of the pairs. This setup proved to be very robust. Although I left the company when financial troubles pushed the founders into a messy divorce, one of the founders asked me to come back a few years later after they restructured. The 4540 had been moved to a new colo but otherwise ignored because none of the admins knew Solaris or bothered to check the monitoring systems. When I returned I discovered in the time I was away 4 drives had failed but ZFS/smf happily took two of the failed drives offline and replaced them with the spares. The two other failed drives left their respective pools degraded but functional. Replacing the failed drives was easy once they arrived. Although neglected for years, none of the applications or databases that relied on that server ever suffered any downtime from the drive failures.
- justsomehnguy 3y ago
  > These two descriptions of resiliency simply are not the same. I would obviously not run the pool in a degraded state for any longer than it takes to resilver the degraded mirror. Then you are just wasting 10TBs. I'm firmly in Murphy's camp regarding the storage troubles (hell, I can't find a specific pic of my passed dog which I know I had from 4 (four!) copies of my photos, ffs) and I treat RAID just as a mean to increase availability. In your case, if you really wouldn't run the degraded array until it is resilvered then your availability is excessive in the all cases except: how long it would take you to recover/restore your data in a catastrophic failure? And why do you have only the critical data 3-2-1ed?
  - greggyb 3y ago
    For the third time, my plan for growing the pool is based on being able to swap in new drives to trade them out with one existing drive in current mirror vdevs. As described previously, this allows me to grow two drives at a time and reduce risk of simultaneous failure in a vdev. As I mentioned before, this sort of swapping is not possible without degrading a raidz vdev. As you have brought up, this configuration optimizes resilvering speed, aka minimizing time spent in a degraded state. Resilvering a mirror is faster than resilvering a raidz, as it requires only sequential read from the good drive and sequential write to the new drive. Resilvering a raidz thrashes all drives in the vdev and requires re-calculating parity along the way. Is it more likely that one of three drives in the degraded raidz1 fails during resilvering or that the one drive in the degraded mirror fails? That is a tough question to answer. What if all the drives are the same age? Well then the likelihood is greater that a resilver will cause a cascading failure. Hence my thrice-repeated desire to use the ability to swap new drives into old vdevs to mix ages, and my mixed-brand vdevs, as described in the root of this thread. In short, resilvering is the highest-risk phase in a pool's lifecycle. I am paying an extra drive's worth of storage to optimize around this risk, and also to realize performance benefits we have discussed upthread. Again, I will state that objectively measurable improvements combined with subjective perception of better desktop performance is sufficient for me to claim that the performance improvement matters. My non-critical data is largely recoverable. Things like VM images and installers. These can be re-created and deployed from code that is in the critical portion, or downloaded from the internet. Indeed, my DR plan does not account for broad internet outages concurrent with a drive failure. A lot of the rest of the non-critical data is media that could be re-ripped from physical media (DVDs, Blu-rays). Being recoverable, it is not worth it to me to pay for cloud storage for the rest. Even "wasting" a drive's worth of storage, it is still a lot cheaper per gigabyte-year to ... you know, not pay for that. So, why do you care so much about my specific storage setup? Does it pass muster yet, or am I still someone who is wrong on the internet to you?