6 ms·

Migrating a ZFS Pool from RAIDZ1 to RAIDZ2

27 points by mtlynch 1y ago

mtlynch 1y ago
Author here. Happy to take any questions or feedback about this post.
- danw1979 1y ago
  Not a question, but I just wanted to say thanks for putting this out there. We’ve all been in a tight spot with storage before and it’s really useful to know about new features and approaches that can help you out and to know the limits of what you’re working with. Even though there are commenters criticising this approach, it’s your data and you’re the one who knows best how to handle it. I think you did a decent job of highlighting the risks.
  - mtlynch 1y ago
    Thanks for reading and the kind words! Yeah, I was surprised this post ended up being so controversial.
xmodem 1y ago
> The neat part is that I did it with only three additional 8 TB disks and never transferred my data to external storage. That's neat! I didn't know there was a way to do this while maintaining data redundancy. > Step 1: Borrow one disk to create a RAIDZ2 pool > To begin, I remove one disk from my original RAIDZ1 pool, leaving it in a degraded state. Oh, there isn't. :facepalm:
- mtlynch 1y ago
  What's the issue? I have backups, so I was prepared for data loss if the pool failed. Is this any riskier than recovering from a single disk failure on a RAIDZ1 pool?
  - SirMaster 1y ago
    Then why go through all this trouble at all? 1. Build new RAIDZ2 pool with all your disks that you plan to use. 2. Restore backup to the new pool. I keep a backup too and so this is how I move to a new larger ZPOOL with a new layout. Either you have to do this because you don't have a backup, and so this is risky. Or you don't need to do this because you have a backup and can just build your new pool and restore your data from the backup.
    - mtlynch 1y ago
      I address this in the post: https://mtlynch.io/raidz1-to-raidz2/#step-2-backing-up-my-data https://mtlynch.io/raidz1-to-raidz2/#step-2-backing-up-my-da...
  - xmodem 1y ago
    No issue. Just that the introduction to your post got me excited that I was going to learn how to do something I didn't previously think was possible.
    - mtlynch 1y ago
      Another reader suggested a different way that achieves the same thing but avoids degrading the pool. You keep the raidz1 pool healthy, create a 5x8TB raidz2 pool with two fake disks, migrate the data, and then only pull the disks from the raidz1 pool when the data successfully migrates. Even if a disk dies during reslivering on the raidz2 pool, you still have 3 of your 4 disks in the raidz1 pool.
  - benlivengood 1y ago
    Administrative risk, mostly. If you accidentally wipefs the wrong disk when moving it to the RAIDZ2 pool or offline a real disk instead of a tmp disk or make some other simple mistake then a zpool might be unrecoverable. An alternative would have been to build a RAIDZ2 out of three new disks and two tmp files and then copy the data over and finally offline a RAIDZ1 disk and online it on the RAIDZ2 zpool. Two copies of the data in each pool would require an actual disk failure in both zpools to lose data during the resilvering (even though both were degraded), and when breaking the original zpool to add a replacement disk for the 5th device in the RAIDZ2 pool you'd have to lose two existing disks in that pool to lose data. No criticism; it's cool to be able to do moves with minimal resources and I've also thought about potential ways to upgrade zpools under weird constraints; especially no free SATA ports for example. I had thought about using the four SATA ports to copy data from half the source disks in a RAIDZ2 to half the destination disks of a new RAIDZ2 while the removed disks would still be a functional zpool on their own as a backup. But ultimately I found a cheap extra box and went for a complete upgrade to larger disks with the benefit of keeping the old box as a local mirror. My upgrade was from a single RAIDZ2 of 4x 4TB disks (8 usable) to a RAIDZ1 of 4x 14TB disks (42 usable) because once both pools existed I was comfortable running the old disks in a 12TB RAIDZ1 to receive snapshots from the primary zpool while I had <12TB used. Still requires 2 disks to fail to lose data, and now I do some offsite backups as well (rsync and s3backer) and have a test system to perform upgrades on before the main system.
SirMaster 1y ago
Just buy 1 extra disk and use that, and then keep it as a spare for your next failure IMO.
- mtlynch 1y ago
  Why is that better? Don't you end up in the same state as my solution except you bought an extraneous 18 TB disk?
  - SirMaster 1y ago
    It would be better because then you wouldn't have to degrade your RAIDZ1 and run with 0 redundancy that could fail at any moment.
    - mtlynch 1y ago
      The extra drive would also have zero redundancy. I know it's risk of 1 drive failing vs. any of 3 drives failing, but disk failures follow a bathtub curb, so I'm more worried about a brand new drive failing than 3 healthy drives that have been running successfully for months. With your extra drive solution, I still have to recreate all my datasets and shares, whereas in my solution, they migrated intact, and I still had backups in case of pool failure. I could zfs send into a giant file on the 18 TB drive, but I'd be reticent to do that because it's just an opaque file that I can't verify will successfully restore. Whereas with my solution, I had the two pools running side by side and could verify everything restored successfully onto the new pool before blowing away the old pool.
      - SirMaster 1y ago
        You seem to be fundamentally misunderstanding me I guess because nothing you are writing here makes sense for what I am proposing. Why did you pull a drive from your RAIDZ1 to purposefully degrade it? I am not sure why you keep saying 18TB. Your drives are 8TB. I am suggesting that you should have simply bought another 8TB disk so you wouldn't have to degrade your RAIDZ1.
commandersaki 1y ago
Hi can someone tell me the difference between RAIDZ1 and RAIDZ2 in a practical sense like you were on the field, rather than the theory. I know RAIDZ1 is ZFS variant of RAID5 and RAIDZ2 is ZFS variant of RAID6; I will use all these terms interchangeably because I'm not too interested in the ZFS special sauce here. In the early 2000s, a lot of people were pushing RAID5. Having worked in a hosting / colocation data centre for many years I had witnessed many RAID5 failures. What would happen is an array would degrade, and more often than not a secondary drive will fail due to undue load on the array as part of the degraded status. Also a lot of times failure would happen on the rebuild process because a lot of the HW implementations were flakey -- but again also the undue stress on all drives as you rebuild. This is why I would suggest a RAID10 setup at the time because of the double lucky failure, and more importantly because you can trivially use a software implementation which is much more safe. Also a lot of the motherboards at the time were offering RAID but this was really just a binary blob in the kernel doing software RAID with a facade of making it appear like hardware which fooled a lot of people. Well we've finally done away with hardware / proprietary RAID and we have ZFS, mdadm, etc. I've normally dismissed RAID6/RAIDZ2 because of the parity/rebuild process and concerns of putting undue stress on the drive. But I think maybe this is premature and that I didn't really understand the consequences of a single drive failure versus a double drive failure. So this is kind of what I want to know: 1. When a single drive fails, is there any undue stress on the array, or because the array can pretty much operate unaffected, there's actually no performance degradation until you rebuild the missing drive, and in the case of software it's really just a negligible hit on the CPU if it has to do hashing/erasure coding/etc. I guess the rebuild process is really just the cost of a zfs scrub at this point but at least it is on a healthy array. 2. The good news of RAID6 over RAID10 is you can always survive two drive failure; but I think this is where things are concerning because the rebuild across two drives places a lot of undue stress on the remaining disks, and if any of those disks die then you're shit outta luck. This scenario is much more similar to a single drive failure in a RAID5 failure. But again, I think the rebuild cost is that of a zfs scrub but with the minimal set of disks. So RAIDZ2 would be a much more solid choice over RAID10 right; at least you will always know you can survive a two drive failure?
- StrangeDoctor 1y ago
abrookewood 1y ago
That sparse drive trick is neat: https://forums.truenas.com/t/raidz1-to-raidz2-without-doubling-drives/40718/2?u=mtlynch https://forums.truenas.com/t/raidz1-to-raidz2-without-doubli...
water9 1y ago
this is incredibly stupid. Do not do this. The whole point of having raid is to never be down to zero redundancy. Drives are more likely to fail during the resilvering process as the heads have to traverse the entire magnetic surface, rebuilding the array.
x-complexity 1y ago
...It took me writing out my original question to recognize why X:=(an in-place upgrade from RAIDZ1 to RAIDZ2) is non-trivial to implement. (Original question: Why X hasn't been implemented yet?) https://www.raidz-calculator.com/raidz-types-reference.aspx https://www.raidz-calculator.com/raidz-types-reference.aspx The general difference between the two is RAIDZ2's additional blocks from its 2nd parity function, which seems independent from the 1st. The (ideal) set of operations that would happen would be: 1) Treat existing parity disk as blank without overwriting it 2) Write to chosen parity disks 1 & 2 from new disks (assertion: new disks >= 2) 3) Blank out old parity disk Step (1) would be near-instant, since it involves no writes. (3) either involves removing the block entries on the old parity disk and treating it as blank (near-instant), or writing zeroes to the disk (lengthy). The bulk of the time would be spent on (2). The process of doing this migration would require siloing off writes to another place whilst the new parity disks are being built. Considering that this migration (A) very rarely occurs, and (B) would require a write silo in the interim, devoting time into creating this error-prone task would not be wise.

27 points by mtlynch 1y ago

mtlynch 1y ago

Author here. Happy to take any questions or feedback about this post.

danw1979 1y ago
Not a question, but I just wanted to say thanks for putting this out there. We’ve all been in a tight spot with storage before and it’s really useful to know about new features and approaches that can help you out and to know the limits of what you’re working with. Even though there are commenters criticising this approach, it’s your data and you’re the one who knows best how to handle it. I think you did a decent job of highlighting the risks.
- mtlynch 1y ago
  Thanks for reading and the kind words! Yeah, I was surprised this post ended up being so controversial.

xmodem 1y ago

> The neat part is that I did it with only three additional 8 TB disks and never transferred my data to external storage. That's neat! I didn't know there was a way to do this while maintaining data redundancy. > Step 1: Borrow one disk to create a RAIDZ2 pool > To begin, I remove one disk from my original RAIDZ1 pool, leaving it in a degraded state. Oh, there isn't. :facepalm:

mtlynch 1y ago
What's the issue? I have backups, so I was prepared for data loss if the pool failed. Is this any riskier than recovering from a single disk failure on a RAIDZ1 pool?
- SirMaster 1y ago
  Then why go through all this trouble at all? 1. Build new RAIDZ2 pool with all your disks that you plan to use. 2. Restore backup to the new pool. I keep a backup too and so this is how I move to a new larger ZPOOL with a new layout. Either you have to do this because you don't have a backup, and so this is risky. Or you don't need to do this because you have a backup and can just build your new pool and restore your data from the backup.
  - mtlynch 1y ago
    I address this in the post: https://mtlynch.io/raidz1-to-raidz2/#step-2-backing-up-my-data https://mtlynch.io/raidz1-to-raidz2/#step-2-backing-up-my-da...
- xmodem 1y ago
  No issue. Just that the introduction to your post got me excited that I was going to learn how to do something I didn't previously think was possible.
  - mtlynch 1y ago
    Another reader suggested a different way that achieves the same thing but avoids degrading the pool. You keep the raidz1 pool healthy, create a 5x8TB raidz2 pool with two fake disks, migrate the data, and then only pull the disks from the raidz1 pool when the data successfully migrates. Even if a disk dies during reslivering on the raidz2 pool, you still have 3 of your 4 disks in the raidz1 pool.
- benlivengood 1y ago
  Administrative risk, mostly. If you accidentally wipefs the wrong disk when moving it to the RAIDZ2 pool or offline a real disk instead of a tmp disk or make some other simple mistake then a zpool might be unrecoverable. An alternative would have been to build a RAIDZ2 out of three new disks and two tmp files and then copy the data over and finally offline a RAIDZ1 disk and online it on the RAIDZ2 zpool. Two copies of the data in each pool would require an actual disk failure in both zpools to lose data during the resilvering (even though both were degraded), and when breaking the original zpool to add a replacement disk for the 5th device in the RAIDZ2 pool you'd have to lose two existing disks in that pool to lose data. No criticism; it's cool to be able to do moves with minimal resources and I've also thought about potential ways to upgrade zpools under weird constraints; especially no free SATA ports for example. I had thought about using the four SATA ports to copy data from half the source disks in a RAIDZ2 to half the destination disks of a new RAIDZ2 while the removed disks would still be a functional zpool on their own as a backup. But ultimately I found a cheap extra box and went for a complete upgrade to larger disks with the benefit of keeping the old box as a local mirror. My upgrade was from a single RAIDZ2 of 4x 4TB disks (8 usable) to a RAIDZ1 of 4x 14TB disks (42 usable) because once both pools existed I was comfortable running the old disks in a 12TB RAIDZ1 to receive snapshots from the primary zpool while I had <12TB used. Still requires 2 disks to fail to lose data, and now I do some offsite backups as well (rsync and s3backer) and have a test system to perform upgrades on before the main system.

SirMaster 1y ago

Just buy 1 extra disk and use that, and then keep it as a spare for your next failure IMO.

mtlynch 1y ago
Why is that better? Don't you end up in the same state as my solution except you bought an extraneous 18 TB disk?
- SirMaster 1y ago
  It would be better because then you wouldn't have to degrade your RAIDZ1 and run with 0 redundancy that could fail at any moment.
  - mtlynch 1y ago
    The extra drive would also have zero redundancy. I know it's risk of 1 drive failing vs. any of 3 drives failing, but disk failures follow a bathtub curb, so I'm more worried about a brand new drive failing than 3 healthy drives that have been running successfully for months. With your extra drive solution, I still have to recreate all my datasets and shares, whereas in my solution, they migrated intact, and I still had backups in case of pool failure. I could zfs send into a giant file on the 18 TB drive, but I'd be reticent to do that because it's just an opaque file that I can't verify will successfully restore. Whereas with my solution, I had the two pools running side by side and could verify everything restored successfully onto the new pool before blowing away the old pool.
    - SirMaster 1y ago
      You seem to be fundamentally misunderstanding me I guess because nothing you are writing here makes sense for what I am proposing. Why did you pull a drive from your RAIDZ1 to purposefully degrade it? I am not sure why you keep saying 18TB. Your drives are 8TB. I am suggesting that you should have simply bought another 8TB disk so you wouldn't have to degrade your RAIDZ1.

commandersaki 1y ago

Hi can someone tell me the difference between RAIDZ1 and RAIDZ2 in a practical sense like you were on the field, rather than the theory. I know RAIDZ1 is ZFS variant of RAID5 and RAIDZ2 is ZFS variant of RAID6; I will use all these terms interchangeably because I'm not too interested in the ZFS special sauce here. In the early 2000s, a lot of people were pushing RAID5. Having worked in a hosting / colocation data centre for many years I had witnessed many RAID5 failures. What would happen is an array would degrade, and more often than not a secondary drive will fail due to undue load on the array as part of the degraded status. Also a lot of times failure would happen on the rebuild process because a lot of the HW implementations were flakey -- but again also the undue stress on all drives as you rebuild. This is why I would suggest a RAID10 setup at the time because of the double lucky failure, and more importantly because you can trivially use a software implementation which is much more safe. Also a lot of the motherboards at the time were offering RAID but this was really just a binary blob in the kernel doing software RAID with a facade of making it appear like hardware which fooled a lot of people. Well we've finally done away with hardware / proprietary RAID and we have ZFS, mdadm, etc. I've normally dismissed RAID6/RAIDZ2 because of the parity/rebuild process and concerns of putting undue stress on the drive. But I think maybe this is premature and that I didn't really understand the consequences of a single drive failure versus a double drive failure. So this is kind of what I want to know: 1. When a single drive fails, is there any undue stress on the array, or because the array can pretty much operate unaffected, there's actually no performance degradation until you rebuild the missing drive, and in the case of software it's really just a negligible hit on the CPU if it has to do hashing/erasure coding/etc. I guess the rebuild process is really just the cost of a zfs scrub at this point but at least it is on a healthy array. 2. The good news of RAID6 over RAID10 is you can always survive two drive failure; but I think this is where things are concerning because the rebuild across two drives places a lot of undue stress on the remaining disks, and if any of those disks die then you're shit outta luck. This scenario is much more similar to a single drive failure in a RAID5 failure. But again, I think the rebuild cost is that of a zfs scrub but with the minimal set of disks. So RAIDZ2 would be a much more solid choice over RAID10 right; at least you will always know you can survive a two drive failure?

StrangeDoctor 1y ago

abrookewood 1y ago

That sparse drive trick is neat: https://forums.truenas.com/t/raidz1-to-raidz2-without-doubling-drives/40718/2?u=mtlynch https://forums.truenas.com/t/raidz1-to-raidz2-without-doubli...

water9 1y ago

this is incredibly stupid. Do not do this. The whole point of having raid is to never be down to zero redundancy. Drives are more likely to fail during the resilvering process as the heads have to traverse the entire magnetic surface, rebuilding the array.

x-complexity 1y ago

...It took me writing out my original question to recognize why X:=(an in-place upgrade from RAIDZ1 to RAIDZ2) is non-trivial to implement. (Original question: Why X hasn't been implemented yet?) https://www.raidz-calculator.com/raidz-types-reference.aspx https://www.raidz-calculator.com/raidz-types-reference.aspx The general difference between the two is RAIDZ2's additional blocks from its 2nd parity function, which seems independent from the 1st. The (ideal) set of operations that would happen would be: 1) Treat existing parity disk as blank without overwriting it 2) Write to chosen parity disks 1 & 2 from new disks (assertion: new disks >= 2) 3) Blank out old parity disk Step (1) would be near-instant, since it involves no writes. (3) either involves removing the block entries on the old parity disk and treating it as blank (near-instant), or writing zeroes to the disk (lengthy). The bulk of the time would be spent on (2). The process of doing this migration would require siloing off writes to another place whilst the new parity disks are being built. Considering that this migration (A) very rarely occurs, and (B) would require a write silo in the interim, devoting time into creating this error-prone task would not be wise.