4 ms·

Just buy 1 extra disk and use that, and then keep it as a spare for your next failure IMO.

by SirMaster 1y ago

mtlynch 1y ago
Why is that better? Don't you end up in the same state as my solution except you bought an extraneous 18 TB disk?
- SirMaster 1y ago
  It would be better because then you wouldn't have to degrade your RAIDZ1 and run with 0 redundancy that could fail at any moment.
  - mtlynch 1y ago
    The extra drive would also have zero redundancy. I know it's risk of 1 drive failing vs. any of 3 drives failing, but disk failures follow a bathtub curb, so I'm more worried about a brand new drive failing than 3 healthy drives that have been running successfully for months. With your extra drive solution, I still have to recreate all my datasets and shares, whereas in my solution, they migrated intact, and I still had backups in case of pool failure. I could zfs send into a giant file on the 18 TB drive, but I'd be reticent to do that because it's just an opaque file that I can't verify will successfully restore. Whereas with my solution, I had the two pools running side by side and could verify everything restored successfully onto the new pool before blowing away the old pool.
    - SirMaster 1y ago
      You seem to be fundamentally misunderstanding me I guess because nothing you are writing here makes sense for what I am proposing. Why did you pull a drive from your RAIDZ1 to purposefully degrade it? I am not sure why you keep saying 18TB. Your drives are 8TB. I am suggesting that you should have simply bought another 8TB disk so you wouldn't have to degrade your RAIDZ1.
      - mtlynch 1y ago
        Oh, I see. I thought you meant I should buy an 18 TB drive, move my data there, blow up my pool, create a RAIDZ2 pool, then move my data back. Yes, I agree I could have reduced the risk of pool failure if I bought an extra 8 TB disk and not degraded the pool. So, it came down to do I for sure spend an extra $120 on an extra drive or do I just take the <1% chance that one of my three other disks fails in a 6-hour window while I'm migrating data. I took the chance, but I also had my data backed up in cloud storage at the file level in case there was a pool failure. In other words, it wasn't worth $120 to me to avoid a <1% risk that I'd have 8ish hours of hassle of recovering from cloud backup after a pool failure.
        fn-mote 1y ago
        Really enjoyed the write up. I have to point out that the surprise risk of the $300 bill from Wasabi dwarfs the cost of the 8TB extra disk. In retrospect, I would have paid the money for the lowered risk, but everybody has a different tolerance for that. Again, great work and very detailed.
        mtlynch 1y ago
        Thanks, I appreciate it!
- burnt-resistor 1y ago
  Raidz2 (survives 2 lost drives) is slower and more complicated than raidz (survives 1 drive loss). The advantage of keeping a powered-off spare (that's good) is there zero wear and (near) zero power usage. The odds of 2 drive failures in rapid succession without temperature rise or SMART issues is around similar odds of a jet turbine breaking down. Raidz3 (survives 3 lost drives) is pure insanity in terms of computational demands. Better off splitting up pools, using multiple systems, or using multiple copy systems up the stack like with Ceph on multiple, simple RAID1 or RAID10 storage nodes.
toast0 1y ago
I found this (and the subsequent discussion) super confusing, so let me restate your plan, and see if this is what you meant? OP described going from 4x disks, add 3x, end up with a 7x raidz2 pool. The steps are 4/4 (steal disk, set up new pool) -> 3/4 + 4/5 (copy and destroy old pool)-> 4/5 (add missing disk) -> 5/5 -> 7/7. Your suggestion is to add 4x disks, so you can do 4/4 (set up new pool) -> 4/4 + 4/5 (copy and destroy old pool) -> 4/5 (add missing disk from old pool) -> 5/5 -> 7/7 + spare (hot or cold?).
- SirMaster 1y ago
  Yes, that's what I would do. And I think it's nice to have a cold spare around to get a degraded pool back to full operation more quickly.
burnt-resistor 1y ago
RAID10 (mdraid+lvm+xfs (never use lvm raid) or btrfs) is way more convenient in terms of rebuild speed, simplicity, performance, and also supports online growing and online shrinking (btrfs only). If there are any failures in a batch, it signals the need for possible proactive replacement. The biggest predictor Google found that SMART doesn't catch is slightly elevated temperatures. ZoL (as opposed to SmartOS/Solaris ZFS) bit me before (array permanently unmountable on good drives) and there was absolutely zero (0) support and they were shameless about it.
- SirMaster 1y ago
  I find that RAIDZ rebuilds almost as fast. Like my 75% full RAIDZ2 pool of 10x14TB disks resilvers a disk in about 18 hours. Which means that it resilvers the 10.5TB of used data onto the new disk at an average of 160MB/s, which is not much slower for the average write speed of the disk by itself. Maybe a mirror would rebuild in like 15 hours? Anyways it doesn't seem like a meaningful amount faster and the much higher capacity of RAIDZ2 over a bunch of mirrors saves a ton of cost both in numbers of disks, but also in slot cost and interface connection costs.
- taskforcegemini 1y ago
  >(never use lvm raid) is that a suggestion or do you meam you never used it? because lvm basically uses mdadm under the hood.
  - burnt-resistor 1y ago
    Don't do that for anything critical. Manage mdraid yourself. It's a whole lot more predictable, manageable, and debuggable than an opaque abstraction. Example: I have 5 XFS volumes across 45 drives on a 4U NAS box with 1 cold spare. Granted, I probably should've chopped them up with a single-box hypervisor RAIDless Ceph cluster of VMs in retrospect.

Just buy 1 extra disk and use that, and then keep it as a spare for your next failure IMO.

mtlynch 1y ago

Why is that better? Don't you end up in the same state as my solution except you bought an extraneous 18 TB disk?

SirMaster 1y ago
It would be better because then you wouldn't have to degrade your RAIDZ1 and run with 0 redundancy that could fail at any moment.
- mtlynch 1y ago
  The extra drive would also have zero redundancy. I know it's risk of 1 drive failing vs. any of 3 drives failing, but disk failures follow a bathtub curb, so I'm more worried about a brand new drive failing than 3 healthy drives that have been running successfully for months. With your extra drive solution, I still have to recreate all my datasets and shares, whereas in my solution, they migrated intact, and I still had backups in case of pool failure. I could zfs send into a giant file on the 18 TB drive, but I'd be reticent to do that because it's just an opaque file that I can't verify will successfully restore. Whereas with my solution, I had the two pools running side by side and could verify everything restored successfully onto the new pool before blowing away the old pool.
  - SirMaster 1y ago
    You seem to be fundamentally misunderstanding me I guess because nothing you are writing here makes sense for what I am proposing. Why did you pull a drive from your RAIDZ1 to purposefully degrade it? I am not sure why you keep saying 18TB. Your drives are 8TB. I am suggesting that you should have simply bought another 8TB disk so you wouldn't have to degrade your RAIDZ1.
    - mtlynch 1y ago
      Oh, I see. I thought you meant I should buy an 18 TB drive, move my data there, blow up my pool, create a RAIDZ2 pool, then move my data back. Yes, I agree I could have reduced the risk of pool failure if I bought an extra 8 TB disk and not degraded the pool. So, it came down to do I for sure spend an extra $120 on an extra drive or do I just take the <1% chance that one of my three other disks fails in a 6-hour window while I'm migrating data. I took the chance, but I also had my data backed up in cloud storage at the file level in case there was a pool failure. In other words, it wasn't worth $120 to me to avoid a <1% risk that I'd have 8ish hours of hassle of recovering from cloud backup after a pool failure.
      - fn-mote 1y ago
        Really enjoyed the write up. I have to point out that the surprise risk of the $300 bill from Wasabi dwarfs the cost of the 8TB extra disk. In retrospect, I would have paid the money for the lowered risk, but everybody has a different tolerance for that. Again, great work and very detailed.
        mtlynch 1y ago
        Thanks, I appreciate it!
burnt-resistor 1y ago
Raidz2 (survives 2 lost drives) is slower and more complicated than raidz (survives 1 drive loss). The advantage of keeping a powered-off spare (that's good) is there zero wear and (near) zero power usage. The odds of 2 drive failures in rapid succession without temperature rise or SMART issues is around similar odds of a jet turbine breaking down. Raidz3 (survives 3 lost drives) is pure insanity in terms of computational demands. Better off splitting up pools, using multiple systems, or using multiple copy systems up the stack like with Ceph on multiple, simple RAID1 or RAID10 storage nodes.

toast0 1y ago

I found this (and the subsequent discussion) super confusing, so let me restate your plan, and see if this is what you meant? OP described going from 4x disks, add 3x, end up with a 7x raidz2 pool. The steps are 4/4 (steal disk, set up new pool) -> 3/4 + 4/5 (copy and destroy old pool)-> 4/5 (add missing disk) -> 5/5 -> 7/7. Your suggestion is to add 4x disks, so you can do 4/4 (set up new pool) -> 4/4 + 4/5 (copy and destroy old pool) -> 4/5 (add missing disk from old pool) -> 5/5 -> 7/7 + spare (hot or cold?).

SirMaster 1y ago
Yes, that's what I would do. And I think it's nice to have a cold spare around to get a degraded pool back to full operation more quickly.

burnt-resistor 1y ago

RAID10 (mdraid+lvm+xfs (never use lvm raid) or btrfs) is way more convenient in terms of rebuild speed, simplicity, performance, and also supports online growing and online shrinking (btrfs only). If there are any failures in a batch, it signals the need for possible proactive replacement. The biggest predictor Google found that SMART doesn't catch is slightly elevated temperatures. ZoL (as opposed to SmartOS/Solaris ZFS) bit me before (array permanently unmountable on good drives) and there was absolutely zero (0) support and they were shameless about it.

SirMaster 1y ago
I find that RAIDZ rebuilds almost as fast. Like my 75% full RAIDZ2 pool of 10x14TB disks resilvers a disk in about 18 hours. Which means that it resilvers the 10.5TB of used data onto the new disk at an average of 160MB/s, which is not much slower for the average write speed of the disk by itself. Maybe a mirror would rebuild in like 15 hours? Anyways it doesn't seem like a meaningful amount faster and the much higher capacity of RAIDZ2 over a bunch of mirrors saves a ton of cost both in numbers of disks, but also in slot cost and interface connection costs.
taskforcegemini 1y ago
>(never use lvm raid) is that a suggestion or do you meam you never used it? because lvm basically uses mdadm under the hood.
- burnt-resistor 1y ago
  Don't do that for anything critical. Manage mdraid yourself. It's a whole lot more predictable, manageable, and debuggable than an opaque abstraction. Example: I have 5 XFS volumes across 45 drives on a 4U NAS box with 1 cold spare. Granted, I probably should've chopped them up with a single-box hypervisor RAIDless Ceph cluster of VMs in retrospect.