6 ms·

Big Kudos to ZFS (on Linux), it's just amazing how sane and stable it is. It saved my ass multiple times in the past years. I've also just finished upgrading a

Big Kudos to ZFS (on Linux), it's just amazing how sane and stable it is. It saved my ass multiple times in the past years. I've also just finished upgrading a RAIDZ1 vdev by replacing one disk at a time with bigger ones. Resilvering took 15h for each disk and there was some trouble with a (already replaced) disk failing during that. Panic mode set in, but ZFS provided the means to fix it quite easily - all good. Best decision ever to pick ZFS.

layoutIfNeeded 7y ago
Replacing one disk at a time in a ZFS pool that's almost full is a bad idea, because all new writes will go to the sole new disk.
- mbreese 7y ago
  But that's the only way to make an existing pool bigger, isn't it? Replace each disk, one at a time, and then auto-expand the pool at the end. The usable space doesn't change until the pool expands at the end. I think you're thinking about adding a new vdev to an existing pool to make the pool larger. This is almost always a bad idea, because like you said, the writes all end up on the new vdev.
  - gravypod 7y ago
    If you can have downtime: you could create a new pool, turn off writes to the old pool, snapshot the entire pool into the new pool using the built in snapshot & transfer mechanics, and then kill the old pool. If you're ok with write performance degradation & doing multiple resilvers than what the op did is fine too.
    - tracker1 7y ago
      That assumes you have enough extra spaces to do it and you didn't start full. Ex: just bought a used Dell R710 with 6 bays, it came with 2TB drives... plan is to play around a bit with those... but I don't have an extra, supported controller with enough slots (even if the cover is off) to populate it while I would migrate. In the end, it'll be full of shucked 8tb drives, but pretty sure I'll just start over at that point. My old nas' drives are 6yo in a decade old synology box, and my trust level is failing before any drives do.
      - gravypod 7y ago
        Something that's important to consider is if your drives can handle a resilver. Each resilver is essentially reading the entire disk (with lots of asterisks) while setting up a parallel zfs system to provision your disks and using the transfer I described will only read your systems fs once. You don't even need to use the same system. You can setups a different system, run the transfer, turn off the main system, swap the disks. It's a hastle but it's an option at your disposal.
        tracker1 7y ago
        If I'm using Raid-Z2, isn't there a way to drop the old drive and add a new drive to take its' place?
- radiowave 7y ago
  I don't think that's correct for the case that dvdgsng is describing. IIRC, the RAIDZ vdev won't allow the use of any extra space until all of its drives have been replaced with bigger ones. What you're describing occurs when growing a ZFS pool by adding a new vdev, i.e. extra disks that sit alongside the existing ones. (Edit: Didn't realize that mbreese had already addressed this point in a different reply.)
  - layoutIfNeeded 7y ago
    True, I didn't pay attention that OP was growing the vdev and not the pool.
the8472 7y ago
you can replace multiple disks at a time, assuming you have enough drive slots.
- radiowave 7y ago
  And assuming the array has sufficient redundancy to allow multiple disks to be offline, which a RAIDZ1 vdev doesn't.
  - kogir 7y ago
    If you have space you can “zfs attach” the new drives to make mirror vdevs with the disks they’re replacing, and then “zfs detach” the old drives to break the mirror when done. This is preferable because you’re never exposed to any additional risk of data loss and can replace more disks at once than original pool redundancy would allow.
    - rainbowzootsuit 7y ago
      zpool replace poolname olddisk newdisk Keeps the olddisk online and utilizes it until newdisk has resilvered, then detaches it. It seems equivalent in safety to zpool attach poolname olddisk newdisk Wait for resilver... zpool detach poolname olddisk
    - rincebrain 7y ago
      Not on a raidz vdev you can't. You can't make mirrors out of anything other than existing mirrors or single disk vdevs. (You _can_ run zpool replace on more disks than the pool redundancy has, assuming you don't need to disconnect the old disks to put the new ones in...running zpool replace on two disks in a four-disk raidz1 is perfectly legal, as long as the old disks are still there.)
  - the8472 7y ago
    zfs replace does not require you to offline the drives. TFA only suggests it as a workaround for the reslivering being slowed down by a faulty drive.
- 7y ago

layoutIfNeeded 7y ago

Replacing one disk at a time in a ZFS pool that's almost full is a bad idea, because all new writes will go to the sole new disk.

mbreese 7y ago
But that's the only way to make an existing pool bigger, isn't it? Replace each disk, one at a time, and then auto-expand the pool at the end. The usable space doesn't change until the pool expands at the end. I think you're thinking about adding a new vdev to an existing pool to make the pool larger. This is almost always a bad idea, because like you said, the writes all end up on the new vdev.
- gravypod 7y ago
  If you can have downtime: you could create a new pool, turn off writes to the old pool, snapshot the entire pool into the new pool using the built in snapshot & transfer mechanics, and then kill the old pool. If you're ok with write performance degradation & doing multiple resilvers than what the op did is fine too.
  - tracker1 7y ago
    That assumes you have enough extra spaces to do it and you didn't start full. Ex: just bought a used Dell R710 with 6 bays, it came with 2TB drives... plan is to play around a bit with those... but I don't have an extra, supported controller with enough slots (even if the cover is off) to populate it while I would migrate. In the end, it'll be full of shucked 8tb drives, but pretty sure I'll just start over at that point. My old nas' drives are 6yo in a decade old synology box, and my trust level is failing before any drives do.
    - gravypod 7y ago
      Something that's important to consider is if your drives can handle a resilver. Each resilver is essentially reading the entire disk (with lots of asterisks) while setting up a parallel zfs system to provision your disks and using the transfer I described will only read your systems fs once. You don't even need to use the same system. You can setups a different system, run the transfer, turn off the main system, swap the disks. It's a hastle but it's an option at your disposal.
      - tracker1 7y ago
        If I'm using Raid-Z2, isn't there a way to drop the old drive and add a new drive to take its' place?
radiowave 7y ago
I don't think that's correct for the case that dvdgsng is describing. IIRC, the RAIDZ vdev won't allow the use of any extra space until all of its drives have been replaced with bigger ones. What you're describing occurs when growing a ZFS pool by adding a new vdev, i.e. extra disks that sit alongside the existing ones. (Edit: Didn't realize that mbreese had already addressed this point in a different reply.)
- layoutIfNeeded 7y ago
  True, I didn't pay attention that OP was growing the vdev and not the pool.

the8472 7y ago

you can replace multiple disks at a time, assuming you have enough drive slots.

radiowave 7y ago
And assuming the array has sufficient redundancy to allow multiple disks to be offline, which a RAIDZ1 vdev doesn't.
- kogir 7y ago
  If you have space you can “zfs attach” the new drives to make mirror vdevs with the disks they’re replacing, and then “zfs detach” the old drives to break the mirror when done. This is preferable because you’re never exposed to any additional risk of data loss and can replace more disks at once than original pool redundancy would allow.
  - rainbowzootsuit 7y ago
    zpool replace poolname olddisk newdisk Keeps the olddisk online and utilizes it until newdisk has resilvered, then detaches it. It seems equivalent in safety to zpool attach poolname olddisk newdisk Wait for resilver... zpool detach poolname olddisk
  - rincebrain 7y ago
    Not on a raidz vdev you can't. You can't make mirrors out of anything other than existing mirrors or single disk vdevs. (You _can_ run zpool replace on more disks than the pool redundancy has, assuming you don't need to disconnect the old disks to put the new ones in...running zpool replace on two disks in a four-disk raidz1 is perfectly legal, as long as the old disks are still there.)
- the8472 7y ago
  zfs replace does not require you to offline the drives. TFA only suggests it as a workaround for the reslivering being slowed down by a faulty drive.
7y ago