4 ms·

Why is that better? Don't you end up in the same state as my solution except you bought an extraneous 18 TB disk?

SirMaster 1y ago
It would be better because then you wouldn't have to degrade your RAIDZ1 and run with 0 redundancy that could fail at any moment.
- mtlynch 1y ago
  The extra drive would also have zero redundancy. I know it's risk of 1 drive failing vs. any of 3 drives failing, but disk failures follow a bathtub curb, so I'm more worried about a brand new drive failing than 3 healthy drives that have been running successfully for months. With your extra drive solution, I still have to recreate all my datasets and shares, whereas in my solution, they migrated intact, and I still had backups in case of pool failure. I could zfs send into a giant file on the 18 TB drive, but I'd be reticent to do that because it's just an opaque file that I can't verify will successfully restore. Whereas with my solution, I had the two pools running side by side and could verify everything restored successfully onto the new pool before blowing away the old pool.
  - SirMaster 1y ago
    You seem to be fundamentally misunderstanding me I guess because nothing you are writing here makes sense for what I am proposing. Why did you pull a drive from your RAIDZ1 to purposefully degrade it? I am not sure why you keep saying 18TB. Your drives are 8TB. I am suggesting that you should have simply bought another 8TB disk so you wouldn't have to degrade your RAIDZ1.
    - mtlynch 1y ago
      Oh, I see. I thought you meant I should buy an 18 TB drive, move my data there, blow up my pool, create a RAIDZ2 pool, then move my data back. Yes, I agree I could have reduced the risk of pool failure if I bought an extra 8 TB disk and not degraded the pool. So, it came down to do I for sure spend an extra $120 on an extra drive or do I just take the <1% chance that one of my three other disks fails in a 6-hour window while I'm migrating data. I took the chance, but I also had my data backed up in cloud storage at the file level in case there was a pool failure. In other words, it wasn't worth $120 to me to avoid a <1% risk that I'd have 8ish hours of hassle of recovering from cloud backup after a pool failure.
      - fn-mote 1y ago
        Really enjoyed the write up. I have to point out that the surprise risk of the $300 bill from Wasabi dwarfs the cost of the 8TB extra disk. In retrospect, I would have paid the money for the lowered risk, but everybody has a different tolerance for that. Again, great work and very detailed.
        mtlynch 1y ago
        Thanks, I appreciate it!
burnt-resistor 1y ago
Raidz2 (survives 2 lost drives) is slower and more complicated than raidz (survives 1 drive loss). The advantage of keeping a powered-off spare (that's good) is there zero wear and (near) zero power usage. The odds of 2 drive failures in rapid succession without temperature rise or SMART issues is around similar odds of a jet turbine breaking down. Raidz3 (survives 3 lost drives) is pure insanity in terms of computational demands. Better off splitting up pools, using multiple systems, or using multiple copy systems up the stack like with Ceph on multiple, simple RAID1 or RAID10 storage nodes.

It would be better because then you wouldn't have to degrade your RAIDZ1 and run with 0 redundancy that could fail at any moment.

mtlynch 1y ago
The extra drive would also have zero redundancy. I know it's risk of 1 drive failing vs. any of 3 drives failing, but disk failures follow a bathtub curb, so I'm more worried about a brand new drive failing than 3 healthy drives that have been running successfully for months. With your extra drive solution, I still have to recreate all my datasets and shares, whereas in my solution, they migrated intact, and I still had backups in case of pool failure. I could zfs send into a giant file on the 18 TB drive, but I'd be reticent to do that because it's just an opaque file that I can't verify will successfully restore. Whereas with my solution, I had the two pools running side by side and could verify everything restored successfully onto the new pool before blowing away the old pool.
- SirMaster 1y ago
  You seem to be fundamentally misunderstanding me I guess because nothing you are writing here makes sense for what I am proposing. Why did you pull a drive from your RAIDZ1 to purposefully degrade it? I am not sure why you keep saying 18TB. Your drives are 8TB. I am suggesting that you should have simply bought another 8TB disk so you wouldn't have to degrade your RAIDZ1.
  - mtlynch 1y ago
    Oh, I see. I thought you meant I should buy an 18 TB drive, move my data there, blow up my pool, create a RAIDZ2 pool, then move my data back. Yes, I agree I could have reduced the risk of pool failure if I bought an extra 8 TB disk and not degraded the pool. So, it came down to do I for sure spend an extra $120 on an extra drive or do I just take the <1% chance that one of my three other disks fails in a 6-hour window while I'm migrating data. I took the chance, but I also had my data backed up in cloud storage at the file level in case there was a pool failure. In other words, it wasn't worth $120 to me to avoid a <1% risk that I'd have 8ish hours of hassle of recovering from cloud backup after a pool failure.
    - fn-mote 1y ago
      Really enjoyed the write up. I have to point out that the surprise risk of the $300 bill from Wasabi dwarfs the cost of the 8TB extra disk. In retrospect, I would have paid the money for the lowered risk, but everybody has a different tolerance for that. Again, great work and very detailed.
      - mtlynch 1y ago
        Thanks, I appreciate it!

burnt-resistor 1y ago

Raidz2 (survives 2 lost drives) is slower and more complicated than raidz (survives 1 drive loss). The advantage of keeping a powered-off spare (that's good) is there zero wear and (near) zero power usage. The odds of 2 drive failures in rapid succession without temperature rise or SMART issues is around similar odds of a jet turbine breaking down. Raidz3 (survives 3 lost drives) is pure insanity in terms of computational demands. Better off splitting up pools, using multiple systems, or using multiple copy systems up the stack like with Ceph on multiple, simple RAID1 or RAID10 storage nodes.