4 ms·

For years, I have been convinced that the only reliable storage solution is an array of drives from different suppliers. My primary desktop runs ZFS with 4x10T

For years, I have been convinced that the only reliable storage solution is an array of drives from different suppliers. My primary desktop runs ZFS with 4x10TB HDDs for bulk storage. These are in two mirrored pairs, so I have 20TB usable space. These consist of three different brands. The two same-branded drives are in different mirrors. When it is time to expand, I intend to swap new drives into the existing mirror, so that I will have mirrors with drives of different brands and ages.

drexlspivey 3y ago
> My primary desktop runs ZFS with 4x10TB HDDs for bulk storage. These are in two mirrored pairs, so I have 20TB usable space. You can have 30TB of usable space with SHR or RAID 5
- greggyb 3y ago
  I sure could, with 1. write performance equivalent to the slowest drive in that single vdev, rather than being striped across two vdevs 2. reads that cannot be distributed across the devices 3. riskier resilvers in the face of a failed drive 4. no ability to safely swap in new drives to existing vdevs as I described later in my post 5. less resilience to device failures There are always tradeoffs. I hope this illuminates some that I have considered.
  - justsomehnguy 3y ago
    Go with LVM parity volumes. 1. For sequential it's up to 3x with full stripe writes, up to 3x in random. 2. Reads in a mirror can't be 'distributed' for the sequential access anyway - repositioning would eat all benefits, it's always faster to read further than to seek and read. 3. For 4 drives it's negligible 4. You can even swap in smaller drives, if your total dataset would fit, no hard limit 5. Exactly the same And you are free to use the free space as JBOD, mirror, mirrored mirror or stripe.
    - greggyb 3y ago
      That would be awfully hard to do as I am not using Linux (: 1. Yes, two mirrored vdevs are not the highest performance configuration for four devices, but they beat a single raidz vdev, which was the suggestion from the prior poster. 2. Random reads are more important than sequential for desktop responsiveness, and mirrors can serve different reads from each disk in the mirror, effectively doubling IOPS (minus overheads). 5. I am not sure I understand here. It seems like you're suggesting 3 data drives and a parity drive. This would allow at most one drive loss before data loss. Two mirrors allow at least one drive loss and up to two drive losses. If there were two parity drives, this would allow up to two drive losses without data losses, but it seems this would have to reduce your claim in point (1) to being 2x performance, since two drives would have to be used for parity writes.
      - justsomehnguy 3y ago
        Ouch. 2. Double IOPS sounds good on paper, at 100% it is still to slow, 100 IOPS at best. 5. Wait. Mirrored mirror (RAID11) on 4 drives would net you only 10TB usable. 'Mirrored pair' sounds like an ordinary RAID10 which can tolerate 1 drive loss because it's no longer redundant for half of the data. 'It can tolerate another loss on the still redundant data' is... like pulling out - works most of the time, but some become parents on the first try. Hence RAID5 have the same redundancy tolerance (1 drive loss for losing R.) then, anything other is gambling. The better question - do you have a proper backup to protect from a logical corruption? Even RAID11 wouldn't help from rf -rf in the wrong path.
        greggyb 3y ago
        It is objectively measurable and subjectively it feels better than tests on a single drive for day to day activity. Objective and subjective improvement is good enough for me. Two mirrors == two vdevs, each a mirror of two disks: $ zpool status pool: zroot state: ONLINE scan: scrub repaired 0B in 02:01:39 with 0 errors on Wed Aug 16 14:08:39 2023 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada2p1.eli ONLINE 0 0 0 ada3p1.eli ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ada0p1.eli ONLINE 0 0 0 ada1p1.eli ONLINE 0 0 0 In a best case failure scenario, I can lose one disk from each mirror. In a worst case failure scenario, I can lose one disk. With one parity disk out of four drives (for 30TB usable), in a best case failure scenario, I can lose one disk. In a worst case failure scenario, I can lose one. These two descriptions of resiliency simply are not the same. I would obviously not run the pool in a degraded state for any longer than it takes to resilver the degraded mirror. Assuming disk failures are independent and one disk has already failed: - 2 mirror vdevs: 1/3 chance that a second disk failure causes data loss, 2/3 chance that it affects the other mirror - 4 disks w/ 1 parity: 100% chance that a second disk failure causes data loss. Again, this simply is not the same characterization of resilience. Please do not mistake this for me claiming that running this pool with three disks is anything other than a situation to remediate immediately. I take regular snapshots, and scrub regularly to ensure data integrity. Critical data is stored in at least three places, one of which is not my home.
bradleyjg 3y ago
The only reliable storage solutions are s3, abs, gcs, etc. Nothing I’m doing at home is going to come close.
- greggyb 3y ago
  As an example, one thing I use my storage pool for is streaming movies via Plex (from Blu-rays I own). If these were stored in S3, in addition to paying for that storage monthly, I would have to pay >$2/movie to stream it to my own house, thanks to Amazon's $0.09/GB egress fees. This is on par with what it would cost to rent a movie from any of the major online streaming services that do rentals. There are some things that are worthwhile to store and have available on the network, but not to pay a third party for.
  - bradleyjg 3y ago
    Fair enough but then on the flip side, how bad is it really if you take a data loss? I’m storing baby pictures and the like at s3 glacier.
    - greggyb 3y ago
      It is annoying and time consuming to recover. There is a whole sort of middle area between ephemeral and needing 11 9s of resilience that you seem to be discounting. Perhaps you live in a world where those two extremes dominate. Maybe you can accept that others do not, and not ask strangers to justify their choices to you?

drexlspivey 3y ago

> My primary desktop runs ZFS with 4x10TB HDDs for bulk storage. These are in two mirrored pairs, so I have 20TB usable space. You can have 30TB of usable space with SHR or RAID 5

greggyb 3y ago
I sure could, with 1. write performance equivalent to the slowest drive in that single vdev, rather than being striped across two vdevs 2. reads that cannot be distributed across the devices 3. riskier resilvers in the face of a failed drive 4. no ability to safely swap in new drives to existing vdevs as I described later in my post 5. less resilience to device failures There are always tradeoffs. I hope this illuminates some that I have considered.
- justsomehnguy 3y ago
  Go with LVM parity volumes. 1. For sequential it's up to 3x with full stripe writes, up to 3x in random. 2. Reads in a mirror can't be 'distributed' for the sequential access anyway - repositioning would eat all benefits, it's always faster to read further than to seek and read. 3. For 4 drives it's negligible 4. You can even swap in smaller drives, if your total dataset would fit, no hard limit 5. Exactly the same And you are free to use the free space as JBOD, mirror, mirrored mirror or stripe.
  - greggyb 3y ago
    That would be awfully hard to do as I am not using Linux (: 1. Yes, two mirrored vdevs are not the highest performance configuration for four devices, but they beat a single raidz vdev, which was the suggestion from the prior poster. 2. Random reads are more important than sequential for desktop responsiveness, and mirrors can serve different reads from each disk in the mirror, effectively doubling IOPS (minus overheads). 5. I am not sure I understand here. It seems like you're suggesting 3 data drives and a parity drive. This would allow at most one drive loss before data loss. Two mirrors allow at least one drive loss and up to two drive losses. If there were two parity drives, this would allow up to two drive losses without data losses, but it seems this would have to reduce your claim in point (1) to being 2x performance, since two drives would have to be used for parity writes.
    - justsomehnguy 3y ago
      Ouch. 2. Double IOPS sounds good on paper, at 100% it is still to slow, 100 IOPS at best. 5. Wait. Mirrored mirror (RAID11) on 4 drives would net you only 10TB usable. 'Mirrored pair' sounds like an ordinary RAID10 which can tolerate 1 drive loss because it's no longer redundant for half of the data. 'It can tolerate another loss on the still redundant data' is... like pulling out - works most of the time, but some become parents on the first try. Hence RAID5 have the same redundancy tolerance (1 drive loss for losing R.) then, anything other is gambling. The better question - do you have a proper backup to protect from a logical corruption? Even RAID11 wouldn't help from rf -rf in the wrong path.
      - greggyb 3y ago
        It is objectively measurable and subjectively it feels better than tests on a single drive for day to day activity. Objective and subjective improvement is good enough for me. Two mirrors == two vdevs, each a mirror of two disks: $ zpool status pool: zroot state: ONLINE scan: scrub repaired 0B in 02:01:39 with 0 errors on Wed Aug 16 14:08:39 2023 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada2p1.eli ONLINE 0 0 0 ada3p1.eli ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ada0p1.eli ONLINE 0 0 0 ada1p1.eli ONLINE 0 0 0 In a best case failure scenario, I can lose one disk from each mirror. In a worst case failure scenario, I can lose one disk. With one parity disk out of four drives (for 30TB usable), in a best case failure scenario, I can lose one disk. In a worst case failure scenario, I can lose one. These two descriptions of resiliency simply are not the same. I would obviously not run the pool in a degraded state for any longer than it takes to resilver the degraded mirror. Assuming disk failures are independent and one disk has already failed: - 2 mirror vdevs: 1/3 chance that a second disk failure causes data loss, 2/3 chance that it affects the other mirror - 4 disks w/ 1 parity: 100% chance that a second disk failure causes data loss. Again, this simply is not the same characterization of resilience. Please do not mistake this for me claiming that running this pool with three disks is anything other than a situation to remediate immediately. I take regular snapshots, and scrub regularly to ensure data integrity. Critical data is stored in at least three places, one of which is not my home.

bradleyjg 3y ago

The only reliable storage solutions are s3, abs, gcs, etc. Nothing I’m doing at home is going to come close.

greggyb 3y ago
As an example, one thing I use my storage pool for is streaming movies via Plex (from Blu-rays I own). If these were stored in S3, in addition to paying for that storage monthly, I would have to pay >$2/movie to stream it to my own house, thanks to Amazon's $0.09/GB egress fees. This is on par with what it would cost to rent a movie from any of the major online streaming services that do rentals. There are some things that are worthwhile to store and have available on the network, but not to pay a third party for.
- bradleyjg 3y ago
  Fair enough but then on the flip side, how bad is it really if you take a data loss? I’m storing baby pictures and the like at s3 glacier.
  - greggyb 3y ago
    It is annoying and time consuming to recover. There is a whole sort of middle area between ephemeral and needing 11 9s of resilience that you seem to be discounting. Perhaps you live in a world where those two extremes dominate. Maybe you can accept that others do not, and not ask strangers to justify their choices to you?