14 ms·

OpenZFS – add disks to existing RAIDZ

244 points by shrubble 3y ago

shrubble 3y ago
"This feature allows disks to be added one at a time to a RAID-Z group, expanding its capacity incrementally. This feature is especially useful for small pools (typically with only one RAID-Z group), where there isn't sufficient hardware to add capacity by adding a whole new RAID-Z group (typically doubling the number of disks)." A feature I am excited to see is being added!
lyjia 3y ago
Wow, I've been hearing this has been in the works for a while. I am glad to see it released! My RAIDZ array awaits new disks!
- cassianoleal 3y ago
  What do you mean by released? It hasn't even been merged yet. :)
  - lyjia 3y ago
    Misread the pull request. But this means it is close to release! It has been in the works for some time.
    - cassianoleal 3y ago
      I do hope so! I have 2 drives waiting to be added to my pool. :)
znpy 3y ago
It bothers me so much that zfs is not into mainline linux. I know it’s due to the license incompatibility… :(
- gigatexal 3y ago
  Same. Maybe one day. Though I don’t know what will happen first: it gets mainlined into the kernel or we get HL3
  - Filligree 3y ago
    So long as the kernel developers are actively hostile to ZFS… You will take your Btrfs and you will like it.
    - gigatexal 3y ago
      I’m holding out for bcachefs and still building ZFS via dkms on current kernels like a madman.
      - Filligree 3y ago
        I’m running bcachefs on my desktop right now. It’s promising, but there’s… bugs. Right now only performance oriented ones, that I’ve noticed, but I’d wait a bit longer.
        thrtythreeforty 3y ago
        Can you elaborate on what kind of bugs? I've been following bcachefs casually because I am very excited for its eventual stabilization. But I am very conservative with my main storage array.
        terinjokes 3y ago
        Not GP, but I tried it out a few weeks ago. I experienced it working for about 15-20 minutes, then it stopped doing any I/O to the disks until I rebooted. After it happened a few times in one evening, I switched to something else.
  - deleted 3y ago
    [deleted]
eminence32 3y ago
The big news here seems to be that iXsystems (the company behind FreeNAS/TrueNAS) is sponsoring this work now. This PR supersedes ones that was opened back in 2021 https://github.com/openzfs/zfs/pull/12225#issuecomment-1610169213 https://github.com/openzfs/zfs/pull/12225#issuecomment-16101...
- seltzered_ 3y ago
  Yep, see also https://freebsdfoundation.org/blog/raid-z-expansion-feature-for-zfs/ https://freebsdfoundation.org/blog/raid-z-expansion-feature-... (2022) (Via https://lobste.rs/s/5ahxj1/raid_z_expansion_feature_for_zfs_goes_live https://lobste.rs/s/5ahxj1/raid_z_expansion_feature_for_zfs_... )
- wkat4242 3y ago
  I hope this corporate sponsorship won't go the way that wireguard went. https://arstechnica.com/gadgets/2021/03/buffer-overruns-license-violations-and-bad-code-freebsd-13s-close-call/ https://arstechnica.com/gadgets/2021/03/buffer-overruns-lice... What worries me the most is them doubling down on it after it was shown to be a mess and totally insecure. I imagine to protect the corporate image. But such corporate interests don't belong in FreeBSD. Just admit you hired the wrong guy (or the right guy at the wrong time) and take the hit. Because it wasn't even the company's fault really. The lack of serious review on the freebsd side was also a big one but I find the 'strings attached' the biggest issue myself because it made mitigation of serious issues such a problem. The large amount of corporate kernel work in Linux is why I went for FreeBSD and to see whatever little go so wrong was worrying.
  - SpecialistK 3y ago
    iXsystems are an entirely different beast than Netgate (pfSense). One can trace its legacy to BSD while it was still at Berkley and has worked with many of the most respected names in the BSD space. The other bought a domain name of a fork and used it to post disparaging messages and Hitler "Downfall" memes slandering them. Source: https://www.wipo.int/amc/en/domains/search/text.jsp?case=D2017-1828 https://www.wipo.int/amc/en/domains/search/text.jsp?case=D20... While I have some disagreements with iXsystems' pivot to ZFS-on-Linux offerings and the like, they're not the clowns that Netgate / pfSense are. 10cm to my right are TrueNAS and opnSense boxes - you won't catch me dead using pfSense in my network after the crap they've pulled.
scheme271 3y ago
This has been floating around for 2 years at this point so might be a long while until it gets in. Interesting, QNAP somehow added this feature into the code that their QuTS Hero NASes uses. I'm not sure how solid or tested the QNAP code is but it's solid enough that they're shipping it in production.
- jtriangle 3y ago
  Qnap does recommend a full backup before doing so, which tells me it's not exactly production ready as you and I would think of it.
  - rincebrain 3y ago
    QNAP's source drops are also kind of wild, in that they branched a looooooooong time ago and have been implementing their own versions of features they wanted since, AFAICT.
  - phpisthebest 3y ago
    I would expect any storage array to recommend a full backup any time you are messing with the physical disks. even "production ready" features one would not add, remove, or do anything with the array with out a full backup. "Production" systems should not even be considered production unless you have a backup of them,
atmosx 3y ago
Finally :-)
crote 3y ago
> After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been expanded once to 6-wide, has 4 data to 2 parity). Does anyone know why this is the case? When expanding an array which is getting full this will result in a far smaller capacity gain than desired. Let's assume we are using 5x 10TB disks which are 90% full. Before the process, each disk will contain 5.4TB of data, 3.6TB of parity, and 1TB of free space. After the process and converting it to 6x 10TB, each disk will contain 4.5TB of data, 3TB of parity, and 2.5TB of free space. We can fill this free space with 1.66TB of data and 0.83TB of parity per disk - after which our entire array will contain 36.96TB of data. If we made a new 6-wide Z2 array, it would be able to contain 40TB of data - so adding a disk this way made us lose over 3TB in capacity! Considering the process is already reading and rewriting basically the entire array, why not recalculate the parity as well?
- mustache_kimono 3y ago
  > Does anyone know why this is the case? > Considering the process is already reading and rewriting basically the entire array, why not recalculate the parity as well? IANA expert but my guess is -- because, here, you don't have to modify block pointers, etc. ZFS RAIDZ is not like traditional RAID, as it's not just a sequence of arbitrary bits, data plus parity. RAIDZ stripe width is variable/dynamic, written in blocks (imagine a 128K block, compressed to ~88K), and there is no way to quickly tell where the parity data is within a written block, where the end of any written block is, etc. If you had to instead, modify the block pointers, I'd assume you have to also change each block in the live tree and all dependent (including snapshot) blocks at the same time? That sounds extraordinarily complicated (and this is the data integrity FS!), and much slower, than just blasting through the data, in order. To do what you want, you can do what one could always do -- zfs send/recv between a filesystem between and old and new filesystem.
  - toast0 3y ago
    > To do what you want, you can do what one could always do -- zfs send/recv between a filesystem between and old and new filesystem. Sure, but that involves having enough spare disks, enough places to put them, and enough places to connect them. This way, while the initial expansion is not ideal, it works. If you really need the space gains from a wider distribution, you can do this expansion and then do the make a new copy, then replace old copy with a new copy dance... although that's counter productive if you have snapshots.
jedberg 3y ago
I wish Apple and Oracle would have just sorted things out and made ZFS the main filesystem for the Mac. Way back in the day when they first designed Time Machine, it was supposed to just be a GUI for ZFS snapshots. How cool would it be if we had a great GUI for ZFS (snapshots, volume management, etc.). I could buy a new external disk, add it to a pool, have seamless storage expansion. It would be great. Ah, what could have been.
- coconut08 3y ago
  yeah imagine a world where you could use time machine to backup 500gb parallels volumes where only the diff was stored between snapshots rather than needing to back up the whole 500gb volume every single time.
  - jedberg 3y ago
    Right, that would be nice wouldn't it? As a workaround, you can create a sparsevolume to store your parallels volume. Sparsevolumes are stores in bands, and only bands that change get backed up. It might be slightly more efficient.
    - coconut08 3y ago
      wow that sounds like an interesting solution!
- mustache_kimono 3y ago
  > How cool would it be if we had a great GUI for ZFS (snapshots, volume management, etc.). How cool would it be if we had a great TUI for ZFS... Live in the now: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm
  - SpecialistK 3y ago
    Now that's a smooth segue to self-promo. I respect it.
hinkley 3y ago
I’m frustrated because this feature was mentioned by Schwartz when it was still in beta. I thought a new era of home computing was about to start. It didn’t, and instead we got The Cloud, which feels like decentralization but is in fact massive centralization (organizational, rather than geographical). Some of us think people should be hosting stuff from home, accessible from their mobile devices. But the first and to me one of the biggest hurdles is managing storage. And that requires a storage appliance that is simpler than using a laptop, not requiring the skills of an IT professional. Drobo tried to make a storage appliance, but once you got to the fine print it had the same set of problems that ZFS still does. All professional storage solutions are built on an assumption of symmetry of hardware. I have n identical (except not the same batch?) drives which I will smear files out across. Consumers will never have drive symmetry. That’s a huge expenditure that few can justify, or much afford. My Synology didn’t like most of my old drives so by the time I had a working array I’d spent practically a laptop on it. For a weirdly shaped computer I couldn’t actually use directly. I’m a developer, I can afford it. None of my friends can. Mom definitely can’t. A consumer solution needs to assume drive asymmetry. That day it is first plugged in, it will contain a couple new drives, and every hard drive the consumer can scrounge up from junk drawers - save two: their current backup drive and an extra copy. Once the array (with one open slot) is built and verified, then one of the backups can go into the array for additional space and speed. From then on, the owner will likely buy one or two new drives every year, at whatever price point they’re willing to pay, and swap out the smallest or slowest drive in the array. Meaning the array will always contain 2-3 different generation of hard drives. Never the same speed and never the same capacity. And they expect that if a rebuild fails, some of their data will still be retrievable. Without a professional data recovery company. which rules out all RAID levels except 0, which is nuts. An algorithm that can handle this scenario is consistent hashing. Weighted consistent hashing can handle disparate resources, by assigning more buckets to faster or larger machines. And it can grow and shrink (in a drive array, the two are sequential or simultaneous). Small and old businesses begin to resemble consumer purchasing patterns. They can’t afford a shiny new array all at once. It’s scrounging and piecemeal. So this isn’t strictly about chasing consumers. I thought ZFS was on a similar path, but the delays in sprouting these features make me wonder.
- Osiris 3y ago
Dwedit 3y ago
So is it safe to use btrfs for a basic Raid-1 yet?
- wtallis 3y ago
  Yeah, the non-parity RAID modes have been safe for a pretty long time, as long as you RTFM when something goes wrong instead of assuming the recovery procedures match what you'd expect coming from a background of traditional RAID or how ZFS does it. I've been using RAID-1 (with RAID1c3 for metadata since that feature became available) on a NAS for over a decade now without loss of data despite loss of more drives over the years than the array started out with.
canvascritic 3y ago
this is a really neat addition to raid-z. i recall setting up my zfs pool in the early 2000s and grappling with disk counts because of how rigid expansion was. Good times. this would've made things so much simpler. small nitpick: in the "during expansion" bit, I thought he could have elaborated a touch on restoring the "health of the raidz vdev" part, didn't really follow his reasoning there. but overall, looking forward to this update. nice work.
- mgerdts 3y ago
  I remember in the decade+ before zfs was introduced and in several of the years since then trying to figure out the optimal stripe size for RAID 10 volumes and the optimal stripe size and count for RAID 5 volumes with Solaris Disk Suite, VxVM, Linux md, etc. Fixing bad decisions has typically meant backup twice, test a restore, fixe the volume, and pray one of your backups is good. :) ZFS has generally been easier to go from zero to working well once you accept that life doesn’t need to be so complicated as legacy md, lvm, and fs layers made it. It also has a pretty big footgun that causes me to carefully read the man page before using “zpool add” or “zpool attach”. Vdev removal is a big help for this.
  - canvascritic 3y ago
    oh, the memories of trying to optimize stripe sizes in the pre-zfs days! i can totally relate to the struggles with solaris disk suite and linux md. the pain of fixing poor decisions and the nail-biting backup processes are all too familiar. zfs really did simplify things in many ways. though i agree, it's always a good idea to tread carefully with commands like `zpool add` and `zpool attach`. glad vdev removal has made that process a bit smoother. it's pretty cool to see how storage management has evolved over the years. cheers for the trip down memory lane. :)
ErneX 3y ago
I’m not an expert whatsoever but what I’ve been doing for my NAS is using mirrored VDEVs. Started with one and later on added a couple more drives for a second mirror. Coincidentally one of the drives of my 1st mirror died few days ago after rebooting the host machine for updates and I replaced it today, it’s been resilvering for a while.
- edmundsauto 3y ago
  I’ve read this is suboptimal because you are now stressing the drive that has the only copy of your data to rebuild. what are your thoughts?
  - deadbunny 3y ago
    Mirrored vdevs resilver a lot faster than zX vdevs. Much less chance of the remaining drive dying during a resilver if it takes hours rather than days.
    - edmundsauto 3y ago
      Is the amount of data read/written the same? I'm not clear why wall time is the relevant metric, unless that's the critical driver of failure likelihood. I would have guessed it's not time but bytes.
      - keeperofdakeys 3y ago
        A mirror resilver is a relatively linear and sequential rewrite, so its very fast. Raidz resilvers require lots of random reads and writes across all the drives, and requires waiting for all drives to read data before it can be written to the replaced drive - "herding cats" sounds appropriate here.
        didntcheck 3y ago
        That makes sense. Usually the received wisdom I hear is "RAIDZ is slower due to the parity calculations", which has always seemed dubious to me on modern CPUs, given we can do much more complex compression and encryption at way faster then disk speed (for HDDs, at least)
tambourine_man 3y ago
If you don’t need real-time redundancy, you maybe better served by something like SnapRAID. It’s more flexible, can mismatch disk sizes, performance requirements are much lower, etc.
matheusmoreira 3y ago
So happy to see this. Incremental expansion is extremely important for consumers, homelabs. Now we can gradually expand capacity.
theshrike79 3y ago
Lack of this feature has kept me on Synology and their SHR-system. I can just slap in any drives of any size and it'll just adapt. As long as the new drives go in an empty slot or are bigger than the previous one, I get more space. Zero CLI commands needed. Now get this feature in unraid or TrueNAS and I'm switching.

244 points by shrubble 3y ago

"This feature allows disks to be added one at a time to a RAID-Z group, expanding its capacity incrementally. This feature is especially useful for small pools (typically with only one RAID-Z group), where there isn't sufficient hardware to add capacity by adding a whole new RAID-Z group (typically doubling the number of disks)." A feature I am excited to see is being added!

lyjia 3y ago

Wow, I've been hearing this has been in the works for a while. I am glad to see it released! My RAIDZ array awaits new disks!

cassianoleal 3y ago
What do you mean by released? It hasn't even been merged yet. :)
- lyjia 3y ago
  Misread the pull request. But this means it is close to release! It has been in the works for some time.
  - cassianoleal 3y ago
    I do hope so! I have 2 drives waiting to be added to my pool. :)

znpy 3y ago

It bothers me so much that zfs is not into mainline linux. I know it’s due to the license incompatibility… :(

gigatexal 3y ago
Same. Maybe one day. Though I don’t know what will happen first: it gets mainlined into the kernel or we get HL3
- Filligree 3y ago
  So long as the kernel developers are actively hostile to ZFS… You will take your Btrfs and you will like it.
  - gigatexal 3y ago
    I’m holding out for bcachefs and still building ZFS via dkms on current kernels like a madman.
    - Filligree 3y ago
      I’m running bcachefs on my desktop right now. It’s promising, but there’s… bugs. Right now only performance oriented ones, that I’ve noticed, but I’d wait a bit longer.
      - thrtythreeforty 3y ago
        Can you elaborate on what kind of bugs? I've been following bcachefs casually because I am very excited for its eventual stabilization. But I am very conservative with my main storage array.
        terinjokes 3y ago
        Not GP, but I tried it out a few weeks ago. I experienced it working for about 15-20 minutes, then it stopped doing any I/O to the disks until I rebooted. After it happened a few times in one evening, I switched to something else.
- deleted 3y ago
  [deleted]

eminence32 3y ago

The big news here seems to be that iXsystems (the company behind FreeNAS/TrueNAS) is sponsoring this work now. This PR supersedes ones that was opened back in 2021 https://github.com/openzfs/zfs/pull/12225#issuecomment-1610169213 https://github.com/openzfs/zfs/pull/12225#issuecomment-16101...

seltzered_ 3y ago
Yep, see also https://freebsdfoundation.org/blog/raid-z-expansion-feature-for-zfs/ https://freebsdfoundation.org/blog/raid-z-expansion-feature-... (2022) (Via https://lobste.rs/s/5ahxj1/raid_z_expansion_feature_for_zfs_goes_live https://lobste.rs/s/5ahxj1/raid_z_expansion_feature_for_zfs_... )
wkat4242 3y ago
I hope this corporate sponsorship won't go the way that wireguard went. https://arstechnica.com/gadgets/2021/03/buffer-overruns-license-violations-and-bad-code-freebsd-13s-close-call/ https://arstechnica.com/gadgets/2021/03/buffer-overruns-lice... What worries me the most is them doubling down on it after it was shown to be a mess and totally insecure. I imagine to protect the corporate image. But such corporate interests don't belong in FreeBSD. Just admit you hired the wrong guy (or the right guy at the wrong time) and take the hit. Because it wasn't even the company's fault really. The lack of serious review on the freebsd side was also a big one but I find the 'strings attached' the biggest issue myself because it made mitigation of serious issues such a problem. The large amount of corporate kernel work in Linux is why I went for FreeBSD and to see whatever little go so wrong was worrying.
- SpecialistK 3y ago
  iXsystems are an entirely different beast than Netgate (pfSense). One can trace its legacy to BSD while it was still at Berkley and has worked with many of the most respected names in the BSD space. The other bought a domain name of a fork and used it to post disparaging messages and Hitler "Downfall" memes slandering them. Source: https://www.wipo.int/amc/en/domains/search/text.jsp?case=D2017-1828 https://www.wipo.int/amc/en/domains/search/text.jsp?case=D20... While I have some disagreements with iXsystems' pivot to ZFS-on-Linux offerings and the like, they're not the clowns that Netgate / pfSense are. 10cm to my right are TrueNAS and opnSense boxes - you won't catch me dead using pfSense in my network after the crap they've pulled.

scheme271 3y ago

This has been floating around for 2 years at this point so might be a long while until it gets in. Interesting, QNAP somehow added this feature into the code that their QuTS Hero NASes uses. I'm not sure how solid or tested the QNAP code is but it's solid enough that they're shipping it in production.

jtriangle 3y ago
Qnap does recommend a full backup before doing so, which tells me it's not exactly production ready as you and I would think of it.
- rincebrain 3y ago
  QNAP's source drops are also kind of wild, in that they branched a looooooooong time ago and have been implementing their own versions of features they wanted since, AFAICT.
- phpisthebest 3y ago
  I would expect any storage array to recommend a full backup any time you are messing with the physical disks. even "production ready" features one would not add, remove, or do anything with the array with out a full backup. "Production" systems should not even be considered production unless you have a backup of them,

atmosx 3y ago

Finally :-)

crote 3y ago

> After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been expanded once to 6-wide, has 4 data to 2 parity). Does anyone know why this is the case? When expanding an array which is getting full this will result in a far smaller capacity gain than desired. Let's assume we are using 5x 10TB disks which are 90% full. Before the process, each disk will contain 5.4TB of data, 3.6TB of parity, and 1TB of free space. After the process and converting it to 6x 10TB, each disk will contain 4.5TB of data, 3TB of parity, and 2.5TB of free space. We can fill this free space with 1.66TB of data and 0.83TB of parity per disk - after which our entire array will contain 36.96TB of data. If we made a new 6-wide Z2 array, it would be able to contain 40TB of data - so adding a disk this way made us lose over 3TB in capacity! Considering the process is already reading and rewriting basically the entire array, why not recalculate the parity as well?

mustache_kimono 3y ago
> Does anyone know why this is the case? > Considering the process is already reading and rewriting basically the entire array, why not recalculate the parity as well? IANA expert but my guess is -- because, here, you don't have to modify block pointers, etc. ZFS RAIDZ is not like traditional RAID, as it's not just a sequence of arbitrary bits, data plus parity. RAIDZ stripe width is variable/dynamic, written in blocks (imagine a 128K block, compressed to ~88K), and there is no way to quickly tell where the parity data is within a written block, where the end of any written block is, etc. If you had to instead, modify the block pointers, I'd assume you have to also change each block in the live tree and all dependent (including snapshot) blocks at the same time? That sounds extraordinarily complicated (and this is the data integrity FS!), and much slower, than just blasting through the data, in order. To do what you want, you can do what one could always do -- zfs send/recv between a filesystem between and old and new filesystem.
- toast0 3y ago
  > To do what you want, you can do what one could always do -- zfs send/recv between a filesystem between and old and new filesystem. Sure, but that involves having enough spare disks, enough places to put them, and enough places to connect them. This way, while the initial expansion is not ideal, it works. If you really need the space gains from a wider distribution, you can do this expansion and then do the make a new copy, then replace old copy with a new copy dance... although that's counter productive if you have snapshots.

jedberg 3y ago

I wish Apple and Oracle would have just sorted things out and made ZFS the main filesystem for the Mac. Way back in the day when they first designed Time Machine, it was supposed to just be a GUI for ZFS snapshots. How cool would it be if we had a great GUI for ZFS (snapshots, volume management, etc.). I could buy a new external disk, add it to a pool, have seamless storage expansion. It would be great. Ah, what could have been.

coconut08 3y ago
yeah imagine a world where you could use time machine to backup 500gb parallels volumes where only the diff was stored between snapshots rather than needing to back up the whole 500gb volume every single time.
- jedberg 3y ago
  Right, that would be nice wouldn't it? As a workaround, you can create a sparsevolume to store your parallels volume. Sparsevolumes are stores in bands, and only bands that change get backed up. It might be slightly more efficient.
  - coconut08 3y ago
    wow that sounds like an interesting solution!
mustache_kimono 3y ago
> How cool would it be if we had a great GUI for ZFS (snapshots, volume management, etc.). How cool would it be if we had a great TUI for ZFS... Live in the now: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm
- SpecialistK 3y ago
  Now that's a smooth segue to self-promo. I respect it.

hinkley 3y ago

I’m frustrated because this feature was mentioned by Schwartz when it was still in beta. I thought a new era of home computing was about to start. It didn’t, and instead we got The Cloud, which feels like decentralization but is in fact massive centralization (organizational, rather than geographical). Some of us think people should be hosting stuff from home, accessible from their mobile devices. But the first and to me one of the biggest hurdles is managing storage. And that requires a storage appliance that is simpler than using a laptop, not requiring the skills of an IT professional. Drobo tried to make a storage appliance, but once you got to the fine print it had the same set of problems that ZFS still does. All professional storage solutions are built on an assumption of symmetry of hardware. I have n identical (except not the same batch?) drives which I will smear files out across. Consumers will never have drive symmetry. That’s a huge expenditure that few can justify, or much afford. My Synology didn’t like most of my old drives so by the time I had a working array I’d spent practically a laptop on it. For a weirdly shaped computer I couldn’t actually use directly. I’m a developer, I can afford it. None of my friends can. Mom definitely can’t. A consumer solution needs to assume drive asymmetry. That day it is first plugged in, it will contain a couple new drives, and every hard drive the consumer can scrounge up from junk drawers - save two: their current backup drive and an extra copy. Once the array (with one open slot) is built and verified, then one of the backups can go into the array for additional space and speed. From then on, the owner will likely buy one or two new drives every year, at whatever price point they’re willing to pay, and swap out the smallest or slowest drive in the array. Meaning the array will always contain 2-3 different generation of hard drives. Never the same speed and never the same capacity. And they expect that if a rebuild fails, some of their data will still be retrievable. Without a professional data recovery company. which rules out all RAID levels except 0, which is nuts. An algorithm that can handle this scenario is consistent hashing. Weighted consistent hashing can handle disparate resources, by assigning more buckets to faster or larger machines. And it can grow and shrink (in a drive array, the two are sequential or simultaneous). Small and old businesses begin to resemble consumer purchasing patterns. They can’t afford a shiny new array all at once. It’s scrounging and piecemeal. So this isn’t strictly about chasing consumers. I thought ZFS was on a similar path, but the delays in sprouting these features make me wonder.

Osiris 3y ago

Dwedit 3y ago

So is it safe to use btrfs for a basic Raid-1 yet?

wtallis 3y ago
Yeah, the non-parity RAID modes have been safe for a pretty long time, as long as you RTFM when something goes wrong instead of assuming the recovery procedures match what you'd expect coming from a background of traditional RAID or how ZFS does it. I've been using RAID-1 (with RAID1c3 for metadata since that feature became available) on a NAS for over a decade now without loss of data despite loss of more drives over the years than the array started out with.

canvascritic 3y ago

this is a really neat addition to raid-z. i recall setting up my zfs pool in the early 2000s and grappling with disk counts because of how rigid expansion was. Good times. this would've made things so much simpler. small nitpick: in the "during expansion" bit, I thought he could have elaborated a touch on restoring the "health of the raidz vdev" part, didn't really follow his reasoning there. but overall, looking forward to this update. nice work.

mgerdts 3y ago
I remember in the decade+ before zfs was introduced and in several of the years since then trying to figure out the optimal stripe size for RAID 10 volumes and the optimal stripe size and count for RAID 5 volumes with Solaris Disk Suite, VxVM, Linux md, etc. Fixing bad decisions has typically meant backup twice, test a restore, fixe the volume, and pray one of your backups is good. :) ZFS has generally been easier to go from zero to working well once you accept that life doesn’t need to be so complicated as legacy md, lvm, and fs layers made it. It also has a pretty big footgun that causes me to carefully read the man page before using “zpool add” or “zpool attach”. Vdev removal is a big help for this.
- canvascritic 3y ago
  oh, the memories of trying to optimize stripe sizes in the pre-zfs days! i can totally relate to the struggles with solaris disk suite and linux md. the pain of fixing poor decisions and the nail-biting backup processes are all too familiar. zfs really did simplify things in many ways. though i agree, it's always a good idea to tread carefully with commands like `zpool add` and `zpool attach`. glad vdev removal has made that process a bit smoother. it's pretty cool to see how storage management has evolved over the years. cheers for the trip down memory lane. :)

ErneX 3y ago

I’m not an expert whatsoever but what I’ve been doing for my NAS is using mirrored VDEVs. Started with one and later on added a couple more drives for a second mirror. Coincidentally one of the drives of my 1st mirror died few days ago after rebooting the host machine for updates and I replaced it today, it’s been resilvering for a while.

edmundsauto 3y ago
I’ve read this is suboptimal because you are now stressing the drive that has the only copy of your data to rebuild. what are your thoughts?
- deadbunny 3y ago
  Mirrored vdevs resilver a lot faster than zX vdevs. Much less chance of the remaining drive dying during a resilver if it takes hours rather than days.
  - edmundsauto 3y ago
    Is the amount of data read/written the same? I'm not clear why wall time is the relevant metric, unless that's the critical driver of failure likelihood. I would have guessed it's not time but bytes.
    - keeperofdakeys 3y ago
      A mirror resilver is a relatively linear and sequential rewrite, so its very fast. Raidz resilvers require lots of random reads and writes across all the drives, and requires waiting for all drives to read data before it can be written to the replaced drive - "herding cats" sounds appropriate here.
      - didntcheck 3y ago
        That makes sense. Usually the received wisdom I hear is "RAIDZ is slower due to the parity calculations", which has always seemed dubious to me on modern CPUs, given we can do much more complex compression and encryption at way faster then disk speed (for HDDs, at least)

tambourine_man 3y ago

If you don’t need real-time redundancy, you maybe better served by something like SnapRAID. It’s more flexible, can mismatch disk sizes, performance requirements are much lower, etc.

matheusmoreira 3y ago

So happy to see this. Incremental expansion is extremely important for consumers, homelabs. Now we can gradually expand capacity.

theshrike79 3y ago

Lack of this feature has kept me on Synology and their SHR-system. I can just slap in any drives of any size and it'll just adapt. As long as the new drives go in an empty slot or are bigger than the previous one, I get more space. Zero CLI commands needed. Now get this feature in unraid or TrueNAS and I'm switching.