11 ms·

Raid-Z Expansion Feature for ZFS Goes Live

228 points by chungy 5y ago

ggm 5y ago
Lots of people have wanted this for ages. I managed to cope with spindle replace and resize into new space (larger spindles) but being able to add more discrete devices and get more parity coverage and more space (I may be incorrectly assuming you get better redundancy as well) is great.
- rincebrain 5y ago
  This trick cannot be used to turn an N-disk raidzP into an [any number]-disk raidzP+1, as far as I understand.
- deleted 5y ago
  [deleted]
reincarnate0x14 5y ago
This is great, there has been a demand for this since forever. Enterprise-y people generally didn't care much but the homelab/SMB users end up dealing with it a lot more than might be naively imagined. Always reminds me of when NetApp used to do their arrays in RAID-4 because it made expansion super-fast, just add a new zeroed disk and only had to update the new disk blocks + parity drive on writes. Used to blow our Netware admin's mind as almost nobody else ever used RAID-4 -- I had it as an interview question along with "what is virtual memory" because you'd get interesting answers :)
- chasil 5y ago
  Can we have "rebalance?" After that, can we have defragmentation? Let's make the best of both worlds.
  - reincarnate0x14 5y ago
    Is fragmentation a serious issue for you? COW filesystems in general aren't great for use-cases that rewrite blocks frequently (databases usually the poster child) but I've never had much problem with it for more general cases even when the free space fragmentation gets north of 70%. Then again most of the storage I care about performance on is NVME. I can see a conceptual sort of super-scrub that balances a zpool and addresses all that that's not on anyone's radar AFAIK.
    - aidenn0 5y ago
      I maintain an archive of nightly builds that the metaslab table is so large that it doesn't stay in ARC without me forcing metaslab_debug on. Workload is roughly Every night write a lot of files 6/7 nights delete a lot of files (i.e. keep weekly snapshots). It took about 3-5 years of doing this to get to the point where write-performance dropped off the clif and enabling metaslab_debug brought it back.
      - exikyut 5y ago
        Hmmmm, in a home-use setting I currently have a scenario where a) random seek/many-small-files performance has been really had since day 1, I initially suspected old/low-end hardware (i3, 1600MHz RAM) but given that I can do just south of 200MB/s (two-way mirror) I'm kinda staring at ZFS expectantly here b) I've admittedly managed to net myself a fair few pathological way-too-many-files situations from projects and whatnot that I really do need to get to cleaning up Fairly early on I noticed apt performance degraded pretty badly, and long before (b) became a substantial concern it got to the point where installing just about anything would take about 60 seconds to do the "Reading database ..." step. I've been idly curious about tweaking different settings to try and improve performance, but it's mostly been an idle curiosity because I don't have a straightforward way to back out of "oh great now what" edge cases. This has probably been going on for just around a year or two, and with absolutely no context I'd be confident saying write volume isn't a shadow of what you're doing :) so perhaps that particular tunable is... maybe not relevant? Or maybe it is. I'm curious.
        5y ago
- matheusmoreira 5y ago
  > the homelab/SMB users end up dealing with it a lot more than might be naively imagined As a home server administrator, I've wanted this feature for so long. Before this, in order to expand an existing array I'd have to fail every single drive and replace them with new higher capacity ones. The only question I have is whether it supports expansion with drives of different capacities.
  - Shish2k 5y ago
    > The only question I have is whether it supports expansion with drives of different capacities. I’m also very interested in this - my main reason for sticking with btrfs is that I can use a variety of odd-sized drives, and expand it by adding a new oddly-sized drive...
    - matheusmoreira 5y ago
      Exactly. That way we can create a storage server out of any random drives we have lying around as well as slowly expand its capacity without having to rebuild the whole thing. I think this capability is vital for people who can't immediately spend thousands of dollars on equipment.
- arjvik 5y ago
  Doesn't RAID5 have the exact same property as RAID4? With a healthy array in both cases, the XOR of all disks should be zero. RAID5 has the benefit of not making the parity drive a bottleneck that will fail sooner.
uniqueuid 5y ago
This is great, but an important and little known caveat is that raidz is limited to the iops of one disk. So growing a raidz will at some point have lots of throughput but suffer in small and random reads and writes. At that point, it will be better to grow the pool with additional, separate raidz.
- Osiris 5y ago
  So you setup two raid-z then stripe them for increased performance?
  - unethical_ban 5y ago
    This is how I understand it. The hierarchy is disk < vdev < zpool. Disk is physical. vdev is logical. Purpose: Disk grouping and redundancy. Composition: One or more disks. zpool is logical. Purpose: Higher-level management of one or more vdevs. Composition: It acts like a JBOD. --- zpools can be thought of as "stripes of vdevs". This, in the narrow sense that the failure of any vdev in a zpool is a permanent loss of the entire zpool. All your redundancy in the ZFS ecosystem is via mirrored or RAID'ed vdevs. --- The setup I have heard of that balances performance, redundancy and space is to do what you say: Have a zpool of multiple mirror-type vdevs. You can also stripe at the vdev level, which I would assume has higher performance than having multiple single-disk vdevs in a pool - I'm unaware of the differences at a low level.
  - oakwhiz 5y ago
    That's a common configuration for larger arrays where you probably don't want to make individual RAIDZ arrays too wide on their own. It's functionally similar to RAID 50 or 60. However when the budget is tight you might be incentivized to compromise on performance or reliability, so this new expansion feature really does help in hobbyist or shoestring budget situations where you just want one huge array to maximize usable space while still being able to tolerate some failures. Typically you would buy disks as you go along to try to stretch the budget further as utilization increases.
- Youden 5y ago
  To be fair, I don't think this is little-known. When a write must be acknowledged by multiple devices (as is the case with RAID1 or RAIDZ), a write requires more IOPS.
FullyFunctional 5y ago
I certainly wanted this. I even heckled Bill Moore about it. Having gone through the expansion the old way (replace each drive one at a time with a larger one), this looks a lot simpler. Unfortunately it appears to not work with simple mirror and stripes (~ RAID10) so it will make no difference for me. (Drives are cheap but performance is not -> RAID10).
- arwineap 5y ago
  This looks different than the old way The old way ( as you referenced ) was to replace each disk one by one with a larger one. If I'm understanding this right, and please correct me, this feature will allow you to add a 5th disk to a 4 disk raidz And if I'm right about that, then this feature wouldn't really make sense for RAID10 anyway
  - FullyFunctional 5y ago
    I love ZFS but this is something that just works in btrfs; mirror just means all blocks live in two physical locations. You certainly can do that even with an odd number of drives. However ZFS is more rigid and doesn’t allow flowing blocks like this, nor dynamic defragmentation.
    - deagle50 5y ago
      Would you use raid 5/6 in btrfs?
      - NavinF 5y ago
        The btrfs raid5/6 write hole is still around, if anyone’s wondering. Though it was only recently that btrfs started warning users that it would eat their data: https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Warning-RAID5-RAID6 https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Wa...
        mrlonglong 5y ago
mnd999 5y ago
Since the Linux crowd moved in zfs development seems to have gone from stability to feature feature feature. I’m starting to get a bit concerned that this isn’t going to end well. I really hope I’m wrong.
- ComputerGuru 5y ago
  The ZoL/OpenZFS-implemented features for encryption shipped with a critical flaw that could permanently corrupt the receiving dataset on subsequent raw send. Same bug also with raw send two a pool with a different ashift, also just recently fixed.
- nightfly 5y ago
  These feature feature features are ones people have been asking for _years_.
- p_l 5y ago
  Nothing really changed in development, ZFSonLinux team was actually one of the more conservative in terms of data safety, what changed is that a bunch of things that were really long in the works coincided in reaching maturity. If you want "feature chasing", FreeBSD ZFS TRIM is the ur example. I've read that code end to end... and I'll leave it at this.
- dsr_ 5y ago
  This feature is being developed in FreeBSD, and will become part of the general ZFSonLinux set.
genpfault 5y ago
So is this feature FreeBSD-only? Or will it be integrated into OpenZFS at some point?
- chungy 5y ago
  It is an experimental feature on FreeBSD 14-CURRENT. It will be merged into OpenZFS eventually (and maybe backported to FreeBSD 13-STABLE and whatever new point releases happen).
  - rincebrain 5y ago
    Where do you see it in CURRENT? AFAICT FreeBSD main is still more or less just a sync against OpenZFS. AFAIK it's being developed against FBSD, but is not any more integrated there than it is in the main OpenZFS project.
    - chungy 5y ago
      Fair point, I haven't actually looked at CURRENT, it's an assumption from the "goes live" part of the article :)
2OEH8eoCRo0 5y ago
- sleepycatgirl 5y ago
  Nah, ZFS is pretty comfy FS, it has lots nice features, it is reasonably fast, and it is stable. And as far as I know, it has been used for fairly long time.
aniou 5y ago
I'm sorry to say that but this article is not entirely true - an illustration "how does traditional raid 4/5/6 do it?" shows ONLY RAID 4. There is a big difference between RAID 4 and RAID 5/6 and former was abandoned a years (decades?) ago in favor of RAID 5 and - later - 6. Of course, it gives "better publicity" for RAID-Z, but it is rather an marketing trick not engineering. See https://en.wikipedia.org/wiki/Standard_RAID_levels https://en.wikipedia.org/wiki/Standard_RAID_levels
- mrighele 5y ago
  Note that the article talks about the way the array is expanded, not how the specific level works. In other words, what they are saying is that the traditional way to expand an array is essentially to rewrite the whole array from scratch, so if the old array has three stripes, with blocks [1,2,3,p1] [4,5,6,p2] and [7,8,9,p3] (with p1 and p2 being the parity blocks), the new array will have stripes [1,2,3,4,p1'], [5,6,7,8,p2'] and [9,x,x,x,p3'], i.e. not only has to move the blocks around, but also recompute essentially all the parity blocks. IF I understand the ZFS approach correctly, the existing blocks are not restructured but only reshuffled, so the new layout will be logically still [1,2,3,p1] [4,5,6,p2] and [7,8,9,p3] but distributed on five disks so [1,2,3,p1,4] [5,6,p2,7,8], [9,p3,x,x,x] It seems that this means less work while expanding, but some space lost unless one manually copies old data in a new place. IF I got it right, I am not sure who is the intended audience for this feature: enterprise users will probably not use it, and power users would probably benefit from getting all the space they could get from the extra disk
  - Elhana 5y ago
    Power users would like to get all the space, but when the choice is either you buy just one hdd and get some space or 4+ hdds to replace old array with a completely new one and then left over with unused old one - most would pick first option.
mrighele 5y ago
For those that like videos more than text, there is a youtube video from last year [1] that explain the feature (unless it's changed since ,but it seems not to be the case). One downside that I see of this approach, if I understand it correctly, is that the data already present on disk will not take advantage of the extra disk per slice. For example, if I have a raidz of 4 disks (so 25% of space "wasted"), and add another disk, new data will be distributed on 5 disks (so 20% of space "wasted") but the old data will keep using stripes of 4 blocks, they will just be reshuffled between the disks. Do I understand it correctly ? [1] https://www.youtube.com/watch?v=yF2KgQGmUic https://www.youtube.com/watch?v=yF2KgQGmUic
- GauntletWizard 5y ago
  That is correct. There are planned solutions for rewriting all your data, though they won't play nicely with one of ZFS's other most important features - Snapshots. The current plan is basically to have a nice userspace utility that will rewrite all your data in-place, but that will cause you to rereplicate everything over your snapshots.
  - mrighele 5y ago
    Thanks for your answer. The point about the snapshots is a very good one, for some reasons I didn't think about it. Rewriting data in place can be tricky, if you have old enough snapshots the newly added space may not be enough. I hope they will find a good enough solution.
- rincebrain 5y ago
  Yes, old data keeps the old data:parity ratio unless you do something like cp or zfs send|recv it and destroy the old copy.
vanillax 5y ago
- 7steps2much 5y ago
  To be fair, they didn't really understand how ZFS works and failed to set up bitrot detection. A "clean" setup would include those, as well as either a messaging system or a regular checkup on how your FS is doing.
  - liuliu 5y ago
    Not only that. A vdev with 15 disks each around 16 (or 18?) TiB with RAID-Z2 just asking for troubles (https://www.zdnet.com/article/why-raid-6-stops-working-in-2019/ https://www.zdnet.com/article/why-raid-6-stops-working-in-20...). Even with 1% disk failures annually (based on Backblaze, for better disks), we are looking at least two disks fail at the same time with 15 disks around 1% chance. With 10 vdevs, that is 10% chance of any of these have failures. They should really move to RAID-Z3 or 8 disks group. With RAID-Z3, we are looking at at least 3 disk fail at the same time with 15 disks around 0.04% chance. With 8 disks, we are looking at 0.2% chance.
    - kalleboo 5y ago
      I've read this article and others like it, and what I don't get is if the URE rates are really that high, why I have I never seen an URE in the bi-weekly scrubs of my pool (184 TB of raw disk), aside from when a single disk was literally going bad?
      - sliken 5y ago
        There's many contributing factors. Power supplies degrade over time, ability to maintain the correct voltages can be impaired for worst case scenarios, like running all disks flat out. Drives (and even more so older drives) can be vibration sensitive, especially in worse case scenarios involving all drives running flat out. With modern manufacturing tolerances being so low, whatever triggered the first disk in your pool to die is likely to get a second in a fairly small window. So sure, the failures are not equally distributed and you've done well with your 184TB pool. Are you tracking the device errors, or only those that are visible to the OS? But multiple disk failures are not particularly uncommon. My experience in this space was two 16 disk servers that I set up 6 5 disk RAID5s (with one global spare per server). Within one month I had 11 of 32 disks die, and barely managed not to lose any user files, and this was not during the 1st month in production. Scary. I've since moved to pairs of servers cross connected to pairs of 60 disk chassis (16 x 12 gbit connections per chassis) with ten 11-disk RAIDz3, 10 global spares, and 6x3.2TB of NVMe cache per server.
gigatexal 5y ago
Amazing! Kudos to the team and everyone behind getting this code into the tree.
Naac 5y ago
I don't understand why the title says "Goes Live"? The code is here[0]. It still needs more testing and cleanup, and will then eventually be merged. After that it'll take some time to make to all the distributions ( freenas, freebsd, etc. ) [0] https://github.com/openzfs/zfs/pull/12225 https://github.com/openzfs/zfs/pull/12225
abrookewood 5y ago
There is a very interesting podcast with Matt Ahrens, co-founder of the ZFS project, that covers RAID-Z expansion as well as the history of ZFS etc. https://changelog.com/podcast/475 https://changelog.com/podcast/475
- fomine3 5y ago
  Thanks. > But the interesting thing about this project is how did it come to be. So a long-requested feature - how did it get funded? So actually, it’s funded by the FreeBSD Foundation. Now I found why OP posted by FreeBSD Foundation.
amatecha 5y ago
Ah man, this is perfect timing. I'm literally in the midst of building a new NAS and intending to use TrueNAS + RAIDZ2. Purposely over-speccing potential drive bay capacity so I can expand later on. I was worried how hard it will be to expand later, but it looks like it may end up being more possible than I thought! Sweeet :)

228 points by chungy 5y ago

ggm 5y ago

Lots of people have wanted this for ages. I managed to cope with spindle replace and resize into new space (larger spindles) but being able to add more discrete devices and get more parity coverage and more space (I may be incorrectly assuming you get better redundancy as well) is great.

rincebrain 5y ago
This trick cannot be used to turn an N-disk raidzP into an [any number]-disk raidzP+1, as far as I understand.
deleted 5y ago
[deleted]

reincarnate0x14 5y ago

This is great, there has been a demand for this since forever. Enterprise-y people generally didn't care much but the homelab/SMB users end up dealing with it a lot more than might be naively imagined. Always reminds me of when NetApp used to do their arrays in RAID-4 because it made expansion super-fast, just add a new zeroed disk and only had to update the new disk blocks + parity drive on writes. Used to blow our Netware admin's mind as almost nobody else ever used RAID-4 -- I had it as an interview question along with "what is virtual memory" because you'd get interesting answers :)

chasil 5y ago
Can we have "rebalance?" After that, can we have defragmentation? Let's make the best of both worlds.
- reincarnate0x14 5y ago
  Is fragmentation a serious issue for you? COW filesystems in general aren't great for use-cases that rewrite blocks frequently (databases usually the poster child) but I've never had much problem with it for more general cases even when the free space fragmentation gets north of 70%. Then again most of the storage I care about performance on is NVME. I can see a conceptual sort of super-scrub that balances a zpool and addresses all that that's not on anyone's radar AFAIK.
  - aidenn0 5y ago
    I maintain an archive of nightly builds that the metaslab table is so large that it doesn't stay in ARC without me forcing metaslab_debug on. Workload is roughly Every night write a lot of files 6/7 nights delete a lot of files (i.e. keep weekly snapshots). It took about 3-5 years of doing this to get to the point where write-performance dropped off the clif and enabling metaslab_debug brought it back.
    - exikyut 5y ago
      Hmmmm, in a home-use setting I currently have a scenario where a) random seek/many-small-files performance has been really had since day 1, I initially suspected old/low-end hardware (i3, 1600MHz RAM) but given that I can do just south of 200MB/s (two-way mirror) I'm kinda staring at ZFS expectantly here b) I've admittedly managed to net myself a fair few pathological way-too-many-files situations from projects and whatnot that I really do need to get to cleaning up Fairly early on I noticed apt performance degraded pretty badly, and long before (b) became a substantial concern it got to the point where installing just about anything would take about 60 seconds to do the "Reading database ..." step. I've been idly curious about tweaking different settings to try and improve performance, but it's mostly been an idle curiosity because I don't have a straightforward way to back out of "oh great now what" edge cases. This has probably been going on for just around a year or two, and with absolutely no context I'd be confident saying write volume isn't a shadow of what you're doing :) so perhaps that particular tunable is... maybe not relevant? Or maybe it is. I'm curious.
      - 5y ago
matheusmoreira 5y ago
> the homelab/SMB users end up dealing with it a lot more than might be naively imagined As a home server administrator, I've wanted this feature for so long. Before this, in order to expand an existing array I'd have to fail every single drive and replace them with new higher capacity ones. The only question I have is whether it supports expansion with drives of different capacities.
- Shish2k 5y ago
  > The only question I have is whether it supports expansion with drives of different capacities. I’m also very interested in this - my main reason for sticking with btrfs is that I can use a variety of odd-sized drives, and expand it by adding a new oddly-sized drive...
  - matheusmoreira 5y ago
    Exactly. That way we can create a storage server out of any random drives we have lying around as well as slowly expand its capacity without having to rebuild the whole thing. I think this capability is vital for people who can't immediately spend thousands of dollars on equipment.
arjvik 5y ago
Doesn't RAID5 have the exact same property as RAID4? With a healthy array in both cases, the XOR of all disks should be zero. RAID5 has the benefit of not making the parity drive a bottleneck that will fail sooner.

uniqueuid 5y ago

This is great, but an important and little known caveat is that raidz is limited to the iops of one disk. So growing a raidz will at some point have lots of throughput but suffer in small and random reads and writes. At that point, it will be better to grow the pool with additional, separate raidz.

Osiris 5y ago
So you setup two raid-z then stripe them for increased performance?
- unethical_ban 5y ago
  This is how I understand it. The hierarchy is disk < vdev < zpool. Disk is physical. vdev is logical. Purpose: Disk grouping and redundancy. Composition: One or more disks. zpool is logical. Purpose: Higher-level management of one or more vdevs. Composition: It acts like a JBOD. --- zpools can be thought of as "stripes of vdevs". This, in the narrow sense that the failure of any vdev in a zpool is a permanent loss of the entire zpool. All your redundancy in the ZFS ecosystem is via mirrored or RAID'ed vdevs. --- The setup I have heard of that balances performance, redundancy and space is to do what you say: Have a zpool of multiple mirror-type vdevs. You can also stripe at the vdev level, which I would assume has higher performance than having multiple single-disk vdevs in a pool - I'm unaware of the differences at a low level.
- oakwhiz 5y ago
  That's a common configuration for larger arrays where you probably don't want to make individual RAIDZ arrays too wide on their own. It's functionally similar to RAID 50 or 60. However when the budget is tight you might be incentivized to compromise on performance or reliability, so this new expansion feature really does help in hobbyist or shoestring budget situations where you just want one huge array to maximize usable space while still being able to tolerate some failures. Typically you would buy disks as you go along to try to stretch the budget further as utilization increases.
Youden 5y ago
To be fair, I don't think this is little-known. When a write must be acknowledged by multiple devices (as is the case with RAID1 or RAIDZ), a write requires more IOPS.

FullyFunctional 5y ago

I certainly wanted this. I even heckled Bill Moore about it. Having gone through the expansion the old way (replace each drive one at a time with a larger one), this looks a lot simpler. Unfortunately it appears to not work with simple mirror and stripes (~ RAID10) so it will make no difference for me. (Drives are cheap but performance is not -> RAID10).

arwineap 5y ago
This looks different than the old way The old way ( as you referenced ) was to replace each disk one by one with a larger one. If I'm understanding this right, and please correct me, this feature will allow you to add a 5th disk to a 4 disk raidz And if I'm right about that, then this feature wouldn't really make sense for RAID10 anyway
- FullyFunctional 5y ago
  I love ZFS but this is something that just works in btrfs; mirror just means all blocks live in two physical locations. You certainly can do that even with an odd number of drives. However ZFS is more rigid and doesn’t allow flowing blocks like this, nor dynamic defragmentation.
  - deagle50 5y ago
    Would you use raid 5/6 in btrfs?
    - NavinF 5y ago
      The btrfs raid5/6 write hole is still around, if anyone’s wondering. Though it was only recently that btrfs started warning users that it would eat their data: https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Warning-RAID5-RAID6 https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Wa...
      - mrlonglong 5y ago

mnd999 5y ago

Since the Linux crowd moved in zfs development seems to have gone from stability to feature feature feature. I’m starting to get a bit concerned that this isn’t going to end well. I really hope I’m wrong.

ComputerGuru 5y ago
The ZoL/OpenZFS-implemented features for encryption shipped with a critical flaw that could permanently corrupt the receiving dataset on subsequent raw send. Same bug also with raw send two a pool with a different ashift, also just recently fixed.
nightfly 5y ago
These feature feature features are ones people have been asking for _years_.
p_l 5y ago
Nothing really changed in development, ZFSonLinux team was actually one of the more conservative in terms of data safety, what changed is that a bunch of things that were really long in the works coincided in reaching maturity. If you want "feature chasing", FreeBSD ZFS TRIM is the ur example. I've read that code end to end... and I'll leave it at this.
dsr_ 5y ago
This feature is being developed in FreeBSD, and will become part of the general ZFSonLinux set.

genpfault 5y ago

So is this feature FreeBSD-only? Or will it be integrated into OpenZFS at some point?

chungy 5y ago
It is an experimental feature on FreeBSD 14-CURRENT. It will be merged into OpenZFS eventually (and maybe backported to FreeBSD 13-STABLE and whatever new point releases happen).
- rincebrain 5y ago
  Where do you see it in CURRENT? AFAICT FreeBSD main is still more or less just a sync against OpenZFS. AFAIK it's being developed against FBSD, but is not any more integrated there than it is in the main OpenZFS project.
  - chungy 5y ago
    Fair point, I haven't actually looked at CURRENT, it's an assumption from the "goes live" part of the article :)

2OEH8eoCRo0 5y ago

sleepycatgirl 5y ago
Nah, ZFS is pretty comfy FS, it has lots nice features, it is reasonably fast, and it is stable. And as far as I know, it has been used for fairly long time.

aniou 5y ago

I'm sorry to say that but this article is not entirely true - an illustration "how does traditional raid 4/5/6 do it?" shows ONLY RAID 4. There is a big difference between RAID 4 and RAID 5/6 and former was abandoned a years (decades?) ago in favor of RAID 5 and - later - 6. Of course, it gives "better publicity" for RAID-Z, but it is rather an marketing trick not engineering. See https://en.wikipedia.org/wiki/Standard_RAID_levels https://en.wikipedia.org/wiki/Standard_RAID_levels

mrighele 5y ago
Note that the article talks about the way the array is expanded, not how the specific level works. In other words, what they are saying is that the traditional way to expand an array is essentially to rewrite the whole array from scratch, so if the old array has three stripes, with blocks [1,2,3,p1] [4,5,6,p2] and [7,8,9,p3] (with p1 and p2 being the parity blocks), the new array will have stripes [1,2,3,4,p1'], [5,6,7,8,p2'] and [9,x,x,x,p3'], i.e. not only has to move the blocks around, but also recompute essentially all the parity blocks. IF I understand the ZFS approach correctly, the existing blocks are not restructured but only reshuffled, so the new layout will be logically still [1,2,3,p1] [4,5,6,p2] and [7,8,9,p3] but distributed on five disks so [1,2,3,p1,4] [5,6,p2,7,8], [9,p3,x,x,x] It seems that this means less work while expanding, but some space lost unless one manually copies old data in a new place. IF I got it right, I am not sure who is the intended audience for this feature: enterprise users will probably not use it, and power users would probably benefit from getting all the space they could get from the extra disk
- Elhana 5y ago
  Power users would like to get all the space, but when the choice is either you buy just one hdd and get some space or 4+ hdds to replace old array with a completely new one and then left over with unused old one - most would pick first option.

mrighele 5y ago

For those that like videos more than text, there is a youtube video from last year [1] that explain the feature (unless it's changed since ,but it seems not to be the case). One downside that I see of this approach, if I understand it correctly, is that the data already present on disk will not take advantage of the extra disk per slice. For example, if I have a raidz of 4 disks (so 25% of space "wasted"), and add another disk, new data will be distributed on 5 disks (so 20% of space "wasted") but the old data will keep using stripes of 4 blocks, they will just be reshuffled between the disks. Do I understand it correctly ? [1] https://www.youtube.com/watch?v=yF2KgQGmUic https://www.youtube.com/watch?v=yF2KgQGmUic

GauntletWizard 5y ago
That is correct. There are planned solutions for rewriting all your data, though they won't play nicely with one of ZFS's other most important features - Snapshots. The current plan is basically to have a nice userspace utility that will rewrite all your data in-place, but that will cause you to rereplicate everything over your snapshots.
- mrighele 5y ago
  Thanks for your answer. The point about the snapshots is a very good one, for some reasons I didn't think about it. Rewriting data in place can be tricky, if you have old enough snapshots the newly added space may not be enough. I hope they will find a good enough solution.
rincebrain 5y ago
Yes, old data keeps the old data:parity ratio unless you do something like cp or zfs send|recv it and destroy the old copy.

vanillax 5y ago

7steps2much 5y ago
To be fair, they didn't really understand how ZFS works and failed to set up bitrot detection. A "clean" setup would include those, as well as either a messaging system or a regular checkup on how your FS is doing.
- liuliu 5y ago
  Not only that. A vdev with 15 disks each around 16 (or 18?) TiB with RAID-Z2 just asking for troubles (https://www.zdnet.com/article/why-raid-6-stops-working-in-2019/ https://www.zdnet.com/article/why-raid-6-stops-working-in-20...). Even with 1% disk failures annually (based on Backblaze, for better disks), we are looking at least two disks fail at the same time with 15 disks around 1% chance. With 10 vdevs, that is 10% chance of any of these have failures. They should really move to RAID-Z3 or 8 disks group. With RAID-Z3, we are looking at at least 3 disk fail at the same time with 15 disks around 0.04% chance. With 8 disks, we are looking at 0.2% chance.
  - kalleboo 5y ago
    I've read this article and others like it, and what I don't get is if the URE rates are really that high, why I have I never seen an URE in the bi-weekly scrubs of my pool (184 TB of raw disk), aside from when a single disk was literally going bad?
    - sliken 5y ago
      There's many contributing factors. Power supplies degrade over time, ability to maintain the correct voltages can be impaired for worst case scenarios, like running all disks flat out. Drives (and even more so older drives) can be vibration sensitive, especially in worse case scenarios involving all drives running flat out. With modern manufacturing tolerances being so low, whatever triggered the first disk in your pool to die is likely to get a second in a fairly small window. So sure, the failures are not equally distributed and you've done well with your 184TB pool. Are you tracking the device errors, or only those that are visible to the OS? But multiple disk failures are not particularly uncommon. My experience in this space was two 16 disk servers that I set up 6 5 disk RAID5s (with one global spare per server). Within one month I had 11 of 32 disks die, and barely managed not to lose any user files, and this was not during the 1st month in production. Scary. I've since moved to pairs of servers cross connected to pairs of 60 disk chassis (16 x 12 gbit connections per chassis) with ten 11-disk RAIDz3, 10 global spares, and 6x3.2TB of NVMe cache per server.

gigatexal 5y ago

Amazing! Kudos to the team and everyone behind getting this code into the tree.

Naac 5y ago

I don't understand why the title says "Goes Live"? The code is here[0]. It still needs more testing and cleanup, and will then eventually be merged. After that it'll take some time to make to all the distributions ( freenas, freebsd, etc. ) [0] https://github.com/openzfs/zfs/pull/12225 https://github.com/openzfs/zfs/pull/12225

abrookewood 5y ago

There is a very interesting podcast with Matt Ahrens, co-founder of the ZFS project, that covers RAID-Z expansion as well as the history of ZFS etc. https://changelog.com/podcast/475 https://changelog.com/podcast/475

fomine3 5y ago
Thanks. > But the interesting thing about this project is how did it come to be. So a long-requested feature - how did it get funded? So actually, it’s funded by the FreeBSD Foundation. Now I found why OP posted by FreeBSD Foundation.

amatecha 5y ago

Ah man, this is perfect timing. I'm literally in the midst of building a new NAS and intending to use TrueNAS + RAIDZ2. Purposely over-speccing potential drive bay capacity so I can expand later on. I was worried how hard it will be to expand later, but it looks like it may end up being more possible than I thought! Sweeet :)