24 ms·

ZFS 2.3 released with ZFS raidz expansion

383 points by scrp 2y ago

cgeier 2y ago
This is huge news for ZFS users (probably mostly those in the hobbyist/home use space, but still). raidz expansion has been one of the most requested features for years.
- jfreax 2y ago
  I'm not yet familiar with zfs and couldn't find it in the release note: Does expansion only works with disk of the same size? Or is adding are bigger/smaller disks possible or do all disk need to have the same size?
  - shiroiushi 2y ago
    As far as I understand, ZFS doesn't work at all with disks of differing sizes (in the same array). So if you try it, it just finds the size of the smallest disk, and uses that for all disks. So if you put an 8TB drive in an array with a bunch of 10TB drives, they'll all be treated as 8TB drives, and the extra 2TB will be ignored on those disks. However, if you replace the smallest disk with a new, larger drive, and resilver, then it'll now use the new smallest disk as the baseline, and use that extra space on the other drives. (Someone please correct me if I'm wrong.)
    - mustache_kimono 2y ago
      > As far as I understand, ZFS doesn't work at all with disks of differing sizes (in the same array). This might be misleading, however, it may only be my understanding of word "array". You can use 2x10TB mirrors as vdev0, and 6x12TB in RAIDZ2 as vdev1 in the same pool/array. You can also stack as many unevenly sized disks as you want in a pool. The actual problem is when you want a different drive topology within a pool or vdev, or you want to mismatch, say, 3 oddly sized drives to create some synthetic redundancy level (2x4TB and 1x8TB to achieve two copies on two disks) like btrfs does/tries to do.
    - tw04 2y ago
      This is the case with any parity based raid, they just hide it or lie to you in various ways. If you have two 6TB dives and two 12TB drives in a single raid-6 array, it is physically impossible to have two drive parity once you exceed 12TB of written capacity. BTRFS and bcachefs can’t magically create more space where none exists on your 6TB drives. They resort to dropping to mirror protection for the excess capacity which you could also do manually with ZFS by giving it partitions instead of the whole drive.
  - chasil 2y ago
    IIRC, you could always replace drives in a raidset with larger devices. When the last drive is replaced, then the new space is recognized. This new operation seems somewhat more sophisticated.
  - zelcon 2y ago
    You need to buy the same exact drive with the same capacity and speed. Your raidz vdev be as small and as slow as your smallest and slowest drive. btrfs and the new bcachefs can do RAID with mixed drives, but I can’t trust either of them with my data yet.
    - Mashimo 2y ago
      > You need to buy the same exact drive AFAIK you can add larger and faster drives, you will just not get any benefits from it.
      - bpye 2y ago
        You can get read speed benefits with faster drives, but your writes will be limited by your slowest.
    - hda111 2y ago
      It doesn't have to be the same exact drive. Mixing drives from different manufacturers (with the same capacity) is often used to prevent correlated failure. ZFS is not using the whole disk, so different disks can be mixed, because the disk often have varying capacity.
    - unixhero 2y ago
      Just have backups. I used btrfs and zfs for different purposes. Never had any lost data or downtime with btrfs since 2016. I only use raid 0 and raid 1 and compression. Btrfs does not havr a hungry ram requirement.
      - tw04 2y ago
        Neither does zfs, that’s a widely repeated red herring from people trying to do dedup in the very early days, and people who misunderstood how it used ram to do caching.
      - zelcon 2y ago
        Tbh the idea of keeping backups defeats the purpose of using RAIDZ (especially RAIDZ3). I don’t want to buy an LTO drive, so if I backup, it’s either buying more HDDs or S3 Glacier ($$$). I like RAIDZ so I don’t have to buy so many drives. I guess it protects you if your house burns down, but how many people do offsite backups for their personal files? And dormant, unpowered HDDs die a lot faster than live, powered HDDs.
  - ryao 2y ago
    You can use different sized disks, but RAID-Z will truncate the space it uses to the lowest common denominator. If you increase the lowest common denominator, RAID-Z should auto-expand to use the additional space. All parity RAID technologies truncate members to the lowest common denominator, rather than just ZFS.
    - GauntletWizard 2y ago
      That's not entirely true, Unraid has mechanisms for unbalanced disks, but they come at a high cost in terms of usability by standard workloads.
      - ryao 2y ago
        Unraid is not a RAID technology: > Unraid saves data to individual drives rather than spreading single files out over multiple drives https://en.wikipedia.org/wiki/Unraid#Software-defined_NAS https://en.wikipedia.org/wiki/Unraid#Software-defined_NAS At least, it is not one in the sense of the original RAID paper that coined the term: https://web.mit.edu/6.033/2015/wwwdocs/papers/Patterson88.pdf https://web.mit.edu/6.033/2015/wwwdocs/papers/Patterson88.pd...
    - wrboyce 2y ago
      Is it definitely the LCD? Given drive of size 15 and 20 the LCD would be 1, no? I had assumed it would just use the size of the smallest drive on every drive (so 15+20->15+15=30). When I first read your comment I was thinking of GCF but even that would be fairly inefficient (GCF(15,20) = 5, so 15+20->5+5=10).
scrp 2y ago
After years in the making ZFS raidz expansaion is finally here. Major features added in release: - RAIDZ Expansion: Add new devices to an existing RAIDZ pool, increasing storage capacity without downtime. - Fast Dedup: A major performance upgrade to the original OpenZFS deduplication functionality. - Direct IO: Allows bypassing the ARC for reads/writes, improving performance in scenarios like NVMe devices where caching may hinder efficiency. - JSON: Optional JSON output for the most used commands. - Long names: Support for file and directory names up to 1023 characters.
- cm2187 2y ago
  But I presume it is still not possible to remove a vdev.
  - mustache_kimono 2y ago
    Is this possible elsewhere (re: other filesystems)?
    - cm2187 2y ago
      It is possible with windows storage space (remove drive from a pool) and mdadm/lvm (remove disk from a RAID array, remove volume from lvm), which to me are the two major alternatives. Don't know about unraid.
      - mustache_kimono 2y ago
        > It is possible with windows storage space (remove drive from a pool) and mdadm/lvm (remove disk from a RAID array, remove volume from lvm), which to me are the two major alternatives. Don't know about unraid. Perhaps I am misunderstanding you, but you can offline and remove drives from a ZFS pool. Do you mean WSS and mdadm/lvm will allow an automatic live rebalance and then reconfigure the drive topology?
        cm2187 2y ago
uniqueuid 2y ago
It's good to see that they were pretty conservative about the expansion. Not only is expansion completely transparent and resumable, it also maintains redundancy throughout the process. That said, there is one tiny caveat people should be aware of: > After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been expanded once to 6-wide, has 4 data to 2 parity).
- chungy 2y ago
  I'm not sure that's really a caveat, it just means old data might be in an inoptimal layout. Even with that, you still get the full benefits of raidzN, where up to N disks can completely fail and the pool will remain functional.
  - stavros 2y ago
    Is that the case? What if I expand a 3-1 array to 3-2? Won't the old blocks remain 3-1?
    - Timshel 2y ago
      I don't believe it supports adding parity drives only data drives.
      - stavros 2y ago
        Ahh interesting, thanks.
        bmicraft 2y ago
        Since preexisting blocks are kept at their current parity ratio and not modified (only redistributed among all devices), increasing the parity level of new blocks won't really be useful in practice anyway.
zelcon 2y ago
Been running it since rc2. It’s insane how long this took to finally ship.
senectus1 2y ago
Would love to use ZFS, but unfortunately Fedora just cant keep up with it...
- vedranm 2y ago
  If you delay upgrading the kernel on occasions, it is more or less fine.
- kawsper 2y ago
  Not sure if it helps you at all, but I have a simple Ruby script that I use to build kernels on Fedora with a specified ZFS version. https://github.com/kaspergrubbe/fedora-kernel-compilation/blob/master/build-kernel.rb https://github.com/kaspergrubbe/fedora-kernel-compilation/bl... It builds on top of the exploded fedora kernel tree, adds zfs and spits out a .rpm that you can install with rpm -ivh. It doesn't play well with dkms because it tries to interfere, so I disable it on my system.
  - _factor 2y ago
    I could never getting working on rpm-ostree distros.
- klauserc 2y ago
  I've been running Fedora on top of the excellent ZFSBootMenu[1] for about a year. You need to pay attention to the kernel versions supported by OpenZFS and might have to wait for support for a couple of weeks. The setup works fine otherwise. [1] https://docs.zfsbootmenu.org https://docs.zfsbootmenu.org
endorphine 2y ago
Can someone describe why they would use ZFS (or similar) for home usage?
- lutorm 2y ago
  Apart from just peace of mind from bitrot, I use it for the snapshotting capability which makes it super easy to do backups. You can snapshot and send the snapshots to other storage with e.g zfs-autobackup and it's trivial and you can't screw it up. If the snapshots exist on the other drive, you know you have a backup.
- vedranm 2y ago
  Several reasons, but major ones (for me) are reliability (checksums and self-healing) and portability (no other modern filesystem can be read and written on Linux, FreeBSD, Windows, and macOS). Snapshots ("boot environments") are also supported by Btrfs (my Linux installations use that so I don't have to worry about having the 3rd party kernel module to read my rootfs). Performance isn't that great either and, assuming Linux, XFS is a better choice if that is your main concern.
- nesarkvechnep 2y ago
  I'm trying to find a reason not to use ZFS at home.
  - dizhn 2y ago
    Requirement for enterprise quality disks, huge RAM (1 gig per TB), ECC, at least x5 disks of redundancy. None of these are things, but people will try to educate you anyway. So use it but keep it to yourself. :)
    - craftkiller 2y ago
      No need to keep it to yourself. As you've mentioned, all of these requirements are misinformation so you can ignore people who repeat them (or even better, tell them to stop spreading misinformation). For those not in the know: You don't need to use enterprise quality disks. There is nothing in the ZFS design that requires enterprise quality disks any more than any other file system. In fact, ZFS has saved my data through multiple consumer-grade HDD failures over the years thanks to raidz. The 1 gig per TB figure is ONLY for when using the ZFS dedup feature, which the ZFS dedup feature is widely regarded as a bad idea except in VERY specific use cases. 99.9% of ZFS users should not and will not use dedup and therefore they do not need ridiculous piles of ram. There is nothing in the design of ZFS any more dangerous to run without ECC than any other filesystem. ECC is a good idea regardless of filesystem but its certainly not a requirement. And you don't need x5 disks of redundancy. It runs great and has benefits even on single-disk systems like laptops. Naturally, having parity drives is better in case a drive fails but on single disk systems you still benefit from the checksumming, snapshotting, boot environments, transparent compression, incremental zfs send/recv, and cross-platform native encryption.
FrostKiwi 2y ago
FINALLY! You can do borderline insane single-vdev setups like RAID-Z3 with 4 disks (3 Disks worth of redundancy) of the most expensive and highest density hard drives money can buy right now, for an initial effective space usage of 25% and then keep buying and expanding Disk by Disk, with the space demand growing, up to something like 12ish disks. Disk prices dropping as time goes on and a spread out failure chance with disks being added at different times.
- uniqueuid 2y ago
  Yes but see my sibling comment. When you expand your array, your existing data will not be stored any more efficiently. To get the new parity/data ratios, you would have to force copies of the data and delete the old, inefficient versions, e.g. with something like this [1] My personal take is that it's a much better idea to buy individual complete raid-z configurations and add new ones / replace old ones (disk by disk!) as you go. [1] https://github.com/markusressel/zfs-inplace-rebalancing https://github.com/markusressel/zfs-inplace-rebalancing
  - Mashimo 2y ago
    I wish something like this would be build into ZFS, so snapshots and current access would not be broken.
    - uniqueuid 2y ago
      True, but I have a gut feeling that a lot of these thorny issues would come up again: https://github.com/openzfs/zfs/issues/3582 https://github.com/openzfs/zfs/issues/3582
averageRoyalty 2y ago
Worth noting that TrueNAS already supports this[0] (I assuming using 2.3.0rc3?). Not sure about the stability, but very exciting. https://www.truenas.com/blog/electric-eel-openzfs-23/ https://www.truenas.com/blog/electric-eel-openzfs-23/
poisonborz 2y ago
I just don't get it how the Windows world - by far the largest PC platform per userbase - still doesn't have any answer to ZFS. Microsoft had WinFS and then ReFS but it's on the backburner and while there is active development (Win11 ships some bits time to time) release is nowhere in sight. There are some lone warriors trying the giant task of creating a ZFS compatibility layer with some projects, but they are far from being mature/usable. How come that Windows still uses a 32 year old file system?
- mustache_kimono 2y ago
  > I just don't get it how the Windows world - by far the largest PC platform per userbase - still doesn't have any answer to ZFS. The mainline Linux kernel doesn't either, and I think the answer is because it's hard and high risk with a return mostly measured in technical respect?
  - ffsm8 2y ago
    Technically speaking, bcachefs has been merged into the Linux Kernel - that makes your initial assertion wrong. But considering it's had two drama events within 1 year of getting merged... I think we can safely confirm your conclusion of it being really hard
    - mustache_kimono 2y ago
      > Technically speaking, bcachefs has been merged into the Linux Kernel - that makes your initial assertion wrong. bcachefs doesn't implement its erasure coding/RAID yet? Doesn't implement send/receive. Doesn't implement scrub/fsck. See: https://bcachefs.org/Roadmap https://bcachefs.org/Roadmap, https://bcachefs.org/Wishlist/ https://bcachefs.org/Wishlist/ btrfs is still more of a legit competitor to ZFS these days and it isn't close to touching ZFS where it matters. If the perpetually half-finished bcachefs and btrfs are the "answer" to ZFS that seems like too little, too late to me.
      - deleted 2y ago
        [deleted]
happosai 2y ago
The annual reminder that if Oracle wanted to contribute positively to the Linux ecosystem, they would update the CDDL license ZFS uses to GPL compatible.
- ryao 2y ago
  This is the annual reply that Oracle cannot change the OpenZFS license because OpenZFS contributors removed the “or any later version” part of the license from their contributions. By the way, comments such as yours seem to assume that Oracle is somehow involved with OpenZFS. Oracle has no connection with OpenZFS outside of owning copyright on the original OpenSolaris sources and a few tiny commits their employees contributed before Oracle purchased Sun. Oracle has its own internal ZFS fork and they have zero interest in bringing it to Linux. They want people to either go on their cloud or buy this: https://www.oracle.com/storage/nas/ https://www.oracle.com/storage/nas/
  - jeroenhd 2y ago
    Is there a reason the OpenZFS contributors don't want to dual-license their code? I'm not too familiar with the CDDL but I'm not sure what advantage it brings to an open source project compared to something like GPL? Having to deal with DKMS is one of the reasons why I'm sticking with BTRFS for doing ZFS-like stuff.
    - ryao 2y ago
      The OpenZFS code is based on the original OpenSolaris code, and the license used is the CDDL because that is what OpenSolaris used. Dual licensing that requires the current OpenSolaris copyright holder to agree. That is unlikely without writing a very big check. Further speculation is not a productive thing to do, but since I know a number of people assume that OpenSolaris copyright holder is the only one preventing this, let me preemptively say that it is not so simple. Different groups have different preferred licenses. Some groups cannot stand certain licenses. Other groups might detest the idea of dual licensing in general since it causes community fragmentation whenever contributors decide to publish changes only under 1 of the 2 licenses. The CDDL was designed to ensure that if Sun Microsystems were acquired by a company hostile to OSS, people could still use Sun’s open source software. In particular, the CDDL has an explicit software patent grant. Some consider that to have been invaluable in preempting lawsuits from a certain company that would rather have ZFS be closed source software.
abrookewood 2y ago
Can someone provide details on this bit please? "Direct IO: Allows bypassing the ARC for reads/writes, improving performance in scenarios like NVMe devices where caching may hinder efficiency". ARC is based in RAM, so how could it reduce performance when used with NVMe devices? They are fast, but they aren't RAM-fast ...
- nolist_policy 2y ago
  Because with a (ARC) cache you have to copy from the app to the cache and then dma to disk. With direct io you can dma directly from the app ram to the disk.
- philjohn 2y ago
  Yes - interested in this too. Is this for both ARC and L2ARC, or just L2ARC?
jakedata 2y ago
Happy to see the ARC bypass for NVMe performance. ZFS really fails to exploit NVMe's potential. Online expansion might be interesting. I tried to use ZFS for some very busy databases and ended up getting bitten badly by the fragmentation bug. The only way to restore performance appears to be copying the data off the volume, nuking it and then copying it back. Now -perhaps- if I expand the zpool then I might be able to reduce fragmentation by copying the tablespace on the same volume.
bitmagier 2y ago
Marvelous!
wkat4242 2y ago
Note: This is online expansion. Expansion was always possible but you did need to take the array down to do it. You could also move to bigger drives but you also had to do that one at a time (and only gain the new capacity once all drives were upgraded of course) As far as I know shrinking a pool is still not possible though. So if you have a pool with 5 drives and add a 6th, you can't go back to 5 drives even if there is very little data in it.
shepherdjerred 2y ago
How does ZFS compare to btrfs? I'm currently using btrfs for my home server, but I've had some strange troubles with it. I'm thinking about switching to ZFS, but I don't want to end up in the same situation.
- ryao 2y ago
  I first tried btrfs 15 years ago with Linux 2.6.33-rc4 if I recall. It developed an unlinkable file within 3 days, so I stopped using it. Later, I found ZFS. It had a few less significant problems, but I was a CS student at the time and I thought I could fix them since they seemed minor in comparison to the issue I had with btrfs, so over the next 18 months, I solved all of the problems that it had that bothered me and sent the patches to be included in the then ZFSOnLinux repository. My effort helped make it production ready on Linux. I have used ZFS ever since and it has worked well for me. If btrfs had been in better shape, I would have been a btrfs contributor. Unfortunately for btrfs, it not only was in bad shape back then, but other btrfs issues continued to bite me every time I tried it over the years for anything serious (e.g. frequent ENOSPC errors when there is still space). ZFS on the other hand just works. Myself and many others did a great deal of work to ensure it works well. The main reason for the difference is that ZFS had a very solid foundation, which was achieved by having some fantastic regression testing facilities. It has a userland version that randomly exercises the code to find bugs before they occur in production and a test suite that is run on every proposed change to help shake out bugs. ZFS also has more people reviewing proposed changes than other filesystems. The Btrfs developers will often state that there is a significant man power difference between the two file systems. I vaguely recall them claiming the difference was a factor of 6. Anyway, few people who use ZFS regret it, so I think you will find you like it too.
- parshimers 2y ago
  btrfs has similar aims to ZFS, but is far less mature. i used it for my root partitions due to it not needing DKMS, but had many troubles. i used it in a fairly simple way, just a mirror. one day, of the drives in the array started to have issues- and btrfs fell on it's face. it remounted everything read-only if i remember correctly, and would not run in degraded mode by default. even mdraid would do better than this without checksumming and so forth. ZFS also likewise, says that the array is faulted, but of course allows it to be used. the fact the default behavior was not RAID, because it's literally missing the R part for reading the data back, made me lose any faith in it. i moved to ZFS and haven't had issues since. there is much more of a community and lots of good tooling around it.

383 points by scrp 2y ago

This is huge news for ZFS users (probably mostly those in the hobbyist/home use space, but still). raidz expansion has been one of the most requested features for years.

jfreax 2y ago
I'm not yet familiar with zfs and couldn't find it in the release note: Does expansion only works with disk of the same size? Or is adding are bigger/smaller disks possible or do all disk need to have the same size?
- shiroiushi 2y ago
  As far as I understand, ZFS doesn't work at all with disks of differing sizes (in the same array). So if you try it, it just finds the size of the smallest disk, and uses that for all disks. So if you put an 8TB drive in an array with a bunch of 10TB drives, they'll all be treated as 8TB drives, and the extra 2TB will be ignored on those disks. However, if you replace the smallest disk with a new, larger drive, and resilver, then it'll now use the new smallest disk as the baseline, and use that extra space on the other drives. (Someone please correct me if I'm wrong.)
  - mustache_kimono 2y ago
    > As far as I understand, ZFS doesn't work at all with disks of differing sizes (in the same array). This might be misleading, however, it may only be my understanding of word "array". You can use 2x10TB mirrors as vdev0, and 6x12TB in RAIDZ2 as vdev1 in the same pool/array. You can also stack as many unevenly sized disks as you want in a pool. The actual problem is when you want a different drive topology within a pool or vdev, or you want to mismatch, say, 3 oddly sized drives to create some synthetic redundancy level (2x4TB and 1x8TB to achieve two copies on two disks) like btrfs does/tries to do.
  - tw04 2y ago
    This is the case with any parity based raid, they just hide it or lie to you in various ways. If you have two 6TB dives and two 12TB drives in a single raid-6 array, it is physically impossible to have two drive parity once you exceed 12TB of written capacity. BTRFS and bcachefs can’t magically create more space where none exists on your 6TB drives. They resort to dropping to mirror protection for the excess capacity which you could also do manually with ZFS by giving it partitions instead of the whole drive.
- chasil 2y ago
  IIRC, you could always replace drives in a raidset with larger devices. When the last drive is replaced, then the new space is recognized. This new operation seems somewhat more sophisticated.
- zelcon 2y ago
  You need to buy the same exact drive with the same capacity and speed. Your raidz vdev be as small and as slow as your smallest and slowest drive. btrfs and the new bcachefs can do RAID with mixed drives, but I can’t trust either of them with my data yet.
  - Mashimo 2y ago
    > You need to buy the same exact drive AFAIK you can add larger and faster drives, you will just not get any benefits from it.
    - bpye 2y ago
      You can get read speed benefits with faster drives, but your writes will be limited by your slowest.
  - hda111 2y ago
    It doesn't have to be the same exact drive. Mixing drives from different manufacturers (with the same capacity) is often used to prevent correlated failure. ZFS is not using the whole disk, so different disks can be mixed, because the disk often have varying capacity.
  - unixhero 2y ago
    Just have backups. I used btrfs and zfs for different purposes. Never had any lost data or downtime with btrfs since 2016. I only use raid 0 and raid 1 and compression. Btrfs does not havr a hungry ram requirement.
    - tw04 2y ago
      Neither does zfs, that’s a widely repeated red herring from people trying to do dedup in the very early days, and people who misunderstood how it used ram to do caching.
    - zelcon 2y ago
      Tbh the idea of keeping backups defeats the purpose of using RAIDZ (especially RAIDZ3). I don’t want to buy an LTO drive, so if I backup, it’s either buying more HDDs or S3 Glacier ($$$). I like RAIDZ so I don’t have to buy so many drives. I guess it protects you if your house burns down, but how many people do offsite backups for their personal files? And dormant, unpowered HDDs die a lot faster than live, powered HDDs.
- ryao 2y ago
  You can use different sized disks, but RAID-Z will truncate the space it uses to the lowest common denominator. If you increase the lowest common denominator, RAID-Z should auto-expand to use the additional space. All parity RAID technologies truncate members to the lowest common denominator, rather than just ZFS.
  - GauntletWizard 2y ago
    That's not entirely true, Unraid has mechanisms for unbalanced disks, but they come at a high cost in terms of usability by standard workloads.
    - ryao 2y ago
      Unraid is not a RAID technology: > Unraid saves data to individual drives rather than spreading single files out over multiple drives https://en.wikipedia.org/wiki/Unraid#Software-defined_NAS https://en.wikipedia.org/wiki/Unraid#Software-defined_NAS At least, it is not one in the sense of the original RAID paper that coined the term: https://web.mit.edu/6.033/2015/wwwdocs/papers/Patterson88.pdf https://web.mit.edu/6.033/2015/wwwdocs/papers/Patterson88.pd...
  - wrboyce 2y ago
    Is it definitely the LCD? Given drive of size 15 and 20 the LCD would be 1, no? I had assumed it would just use the size of the smallest drive on every drive (so 15+20->15+15=30). When I first read your comment I was thinking of GCF but even that would be fairly inefficient (GCF(15,20) = 5, so 15+20->5+5=10).

scrp 2y ago

After years in the making ZFS raidz expansaion is finally here. Major features added in release: - RAIDZ Expansion: Add new devices to an existing RAIDZ pool, increasing storage capacity without downtime. - Fast Dedup: A major performance upgrade to the original OpenZFS deduplication functionality. - Direct IO: Allows bypassing the ARC for reads/writes, improving performance in scenarios like NVMe devices where caching may hinder efficiency. - JSON: Optional JSON output for the most used commands. - Long names: Support for file and directory names up to 1023 characters.

cm2187 2y ago
But I presume it is still not possible to remove a vdev.
- mustache_kimono 2y ago
  Is this possible elsewhere (re: other filesystems)?
  - cm2187 2y ago
    It is possible with windows storage space (remove drive from a pool) and mdadm/lvm (remove disk from a RAID array, remove volume from lvm), which to me are the two major alternatives. Don't know about unraid.
    - mustache_kimono 2y ago
      > It is possible with windows storage space (remove drive from a pool) and mdadm/lvm (remove disk from a RAID array, remove volume from lvm), which to me are the two major alternatives. Don't know about unraid. Perhaps I am misunderstanding you, but you can offline and remove drives from a ZFS pool. Do you mean WSS and mdadm/lvm will allow an automatic live rebalance and then reconfigure the drive topology?
      - cm2187 2y ago

uniqueuid 2y ago

It's good to see that they were pretty conservative about the expansion. Not only is expansion completely transparent and resumable, it also maintains redundancy throughout the process. That said, there is one tiny caveat people should be aware of: > After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been expanded once to 6-wide, has 4 data to 2 parity).

chungy 2y ago
I'm not sure that's really a caveat, it just means old data might be in an inoptimal layout. Even with that, you still get the full benefits of raidzN, where up to N disks can completely fail and the pool will remain functional.
- stavros 2y ago
  Is that the case? What if I expand a 3-1 array to 3-2? Won't the old blocks remain 3-1?
  - Timshel 2y ago
    I don't believe it supports adding parity drives only data drives.
    - stavros 2y ago
      Ahh interesting, thanks.
      - bmicraft 2y ago
        Since preexisting blocks are kept at their current parity ratio and not modified (only redistributed among all devices), increasing the parity level of new blocks won't really be useful in practice anyway.

zelcon 2y ago

Been running it since rc2. It’s insane how long this took to finally ship.

senectus1 2y ago

Would love to use ZFS, but unfortunately Fedora just cant keep up with it...

vedranm 2y ago
If you delay upgrading the kernel on occasions, it is more or less fine.
kawsper 2y ago
Not sure if it helps you at all, but I have a simple Ruby script that I use to build kernels on Fedora with a specified ZFS version. https://github.com/kaspergrubbe/fedora-kernel-compilation/blob/master/build-kernel.rb https://github.com/kaspergrubbe/fedora-kernel-compilation/bl... It builds on top of the exploded fedora kernel tree, adds zfs and spits out a .rpm that you can install with rpm -ivh. It doesn't play well with dkms because it tries to interfere, so I disable it on my system.
- _factor 2y ago
  I could never getting working on rpm-ostree distros.
klauserc 2y ago
I've been running Fedora on top of the excellent ZFSBootMenu[1] for about a year. You need to pay attention to the kernel versions supported by OpenZFS and might have to wait for support for a couple of weeks. The setup works fine otherwise. [1] https://docs.zfsbootmenu.org https://docs.zfsbootmenu.org

endorphine 2y ago

Can someone describe why they would use ZFS (or similar) for home usage?

lutorm 2y ago
Apart from just peace of mind from bitrot, I use it for the snapshotting capability which makes it super easy to do backups. You can snapshot and send the snapshots to other storage with e.g zfs-autobackup and it's trivial and you can't screw it up. If the snapshots exist on the other drive, you know you have a backup.
vedranm 2y ago
Several reasons, but major ones (for me) are reliability (checksums and self-healing) and portability (no other modern filesystem can be read and written on Linux, FreeBSD, Windows, and macOS). Snapshots ("boot environments") are also supported by Btrfs (my Linux installations use that so I don't have to worry about having the 3rd party kernel module to read my rootfs). Performance isn't that great either and, assuming Linux, XFS is a better choice if that is your main concern.
nesarkvechnep 2y ago
I'm trying to find a reason not to use ZFS at home.
- dizhn 2y ago
  Requirement for enterprise quality disks, huge RAM (1 gig per TB), ECC, at least x5 disks of redundancy. None of these are things, but people will try to educate you anyway. So use it but keep it to yourself. :)
  - craftkiller 2y ago
    No need to keep it to yourself. As you've mentioned, all of these requirements are misinformation so you can ignore people who repeat them (or even better, tell them to stop spreading misinformation). For those not in the know: You don't need to use enterprise quality disks. There is nothing in the ZFS design that requires enterprise quality disks any more than any other file system. In fact, ZFS has saved my data through multiple consumer-grade HDD failures over the years thanks to raidz. The 1 gig per TB figure is ONLY for when using the ZFS dedup feature, which the ZFS dedup feature is widely regarded as a bad idea except in VERY specific use cases. 99.9% of ZFS users should not and will not use dedup and therefore they do not need ridiculous piles of ram. There is nothing in the design of ZFS any more dangerous to run without ECC than any other filesystem. ECC is a good idea regardless of filesystem but its certainly not a requirement. And you don't need x5 disks of redundancy. It runs great and has benefits even on single-disk systems like laptops. Naturally, having parity drives is better in case a drive fails but on single disk systems you still benefit from the checksumming, snapshotting, boot environments, transparent compression, incremental zfs send/recv, and cross-platform native encryption.

FrostKiwi 2y ago

FINALLY! You can do borderline insane single-vdev setups like RAID-Z3 with 4 disks (3 Disks worth of redundancy) of the most expensive and highest density hard drives money can buy right now, for an initial effective space usage of 25% and then keep buying and expanding Disk by Disk, with the space demand growing, up to something like 12ish disks. Disk prices dropping as time goes on and a spread out failure chance with disks being added at different times.

uniqueuid 2y ago
Yes but see my sibling comment. When you expand your array, your existing data will not be stored any more efficiently. To get the new parity/data ratios, you would have to force copies of the data and delete the old, inefficient versions, e.g. with something like this [1] My personal take is that it's a much better idea to buy individual complete raid-z configurations and add new ones / replace old ones (disk by disk!) as you go. [1] https://github.com/markusressel/zfs-inplace-rebalancing https://github.com/markusressel/zfs-inplace-rebalancing
- Mashimo 2y ago
  I wish something like this would be build into ZFS, so snapshots and current access would not be broken.
  - uniqueuid 2y ago
    True, but I have a gut feeling that a lot of these thorny issues would come up again: https://github.com/openzfs/zfs/issues/3582 https://github.com/openzfs/zfs/issues/3582

averageRoyalty 2y ago

Worth noting that TrueNAS already supports this[0] (I assuming using 2.3.0rc3?). Not sure about the stability, but very exciting. https://www.truenas.com/blog/electric-eel-openzfs-23/ https://www.truenas.com/blog/electric-eel-openzfs-23/

poisonborz 2y ago

I just don't get it how the Windows world - by far the largest PC platform per userbase - still doesn't have any answer to ZFS. Microsoft had WinFS and then ReFS but it's on the backburner and while there is active development (Win11 ships some bits time to time) release is nowhere in sight. There are some lone warriors trying the giant task of creating a ZFS compatibility layer with some projects, but they are far from being mature/usable. How come that Windows still uses a 32 year old file system?

mustache_kimono 2y ago
> I just don't get it how the Windows world - by far the largest PC platform per userbase - still doesn't have any answer to ZFS. The mainline Linux kernel doesn't either, and I think the answer is because it's hard and high risk with a return mostly measured in technical respect?
- ffsm8 2y ago
  Technically speaking, bcachefs has been merged into the Linux Kernel - that makes your initial assertion wrong. But considering it's had two drama events within 1 year of getting merged... I think we can safely confirm your conclusion of it being really hard
  - mustache_kimono 2y ago
    > Technically speaking, bcachefs has been merged into the Linux Kernel - that makes your initial assertion wrong. bcachefs doesn't implement its erasure coding/RAID yet? Doesn't implement send/receive. Doesn't implement scrub/fsck. See: https://bcachefs.org/Roadmap https://bcachefs.org/Roadmap, https://bcachefs.org/Wishlist/ https://bcachefs.org/Wishlist/ btrfs is still more of a legit competitor to ZFS these days and it isn't close to touching ZFS where it matters. If the perpetually half-finished bcachefs and btrfs are the "answer" to ZFS that seems like too little, too late to me.
    - deleted 2y ago
      [deleted]

happosai 2y ago

The annual reminder that if Oracle wanted to contribute positively to the Linux ecosystem, they would update the CDDL license ZFS uses to GPL compatible.

ryao 2y ago
This is the annual reply that Oracle cannot change the OpenZFS license because OpenZFS contributors removed the “or any later version” part of the license from their contributions. By the way, comments such as yours seem to assume that Oracle is somehow involved with OpenZFS. Oracle has no connection with OpenZFS outside of owning copyright on the original OpenSolaris sources and a few tiny commits their employees contributed before Oracle purchased Sun. Oracle has its own internal ZFS fork and they have zero interest in bringing it to Linux. They want people to either go on their cloud or buy this: https://www.oracle.com/storage/nas/ https://www.oracle.com/storage/nas/
- jeroenhd 2y ago
  Is there a reason the OpenZFS contributors don't want to dual-license their code? I'm not too familiar with the CDDL but I'm not sure what advantage it brings to an open source project compared to something like GPL? Having to deal with DKMS is one of the reasons why I'm sticking with BTRFS for doing ZFS-like stuff.
  - ryao 2y ago
    The OpenZFS code is based on the original OpenSolaris code, and the license used is the CDDL because that is what OpenSolaris used. Dual licensing that requires the current OpenSolaris copyright holder to agree. That is unlikely without writing a very big check. Further speculation is not a productive thing to do, but since I know a number of people assume that OpenSolaris copyright holder is the only one preventing this, let me preemptively say that it is not so simple. Different groups have different preferred licenses. Some groups cannot stand certain licenses. Other groups might detest the idea of dual licensing in general since it causes community fragmentation whenever contributors decide to publish changes only under 1 of the 2 licenses. The CDDL was designed to ensure that if Sun Microsystems were acquired by a company hostile to OSS, people could still use Sun’s open source software. In particular, the CDDL has an explicit software patent grant. Some consider that to have been invaluable in preempting lawsuits from a certain company that would rather have ZFS be closed source software.

abrookewood 2y ago

Can someone provide details on this bit please? "Direct IO: Allows bypassing the ARC for reads/writes, improving performance in scenarios like NVMe devices where caching may hinder efficiency". ARC is based in RAM, so how could it reduce performance when used with NVMe devices? They are fast, but they aren't RAM-fast ...

nolist_policy 2y ago
Because with a (ARC) cache you have to copy from the app to the cache and then dma to disk. With direct io you can dma directly from the app ram to the disk.
philjohn 2y ago
Yes - interested in this too. Is this for both ARC and L2ARC, or just L2ARC?

jakedata 2y ago

Happy to see the ARC bypass for NVMe performance. ZFS really fails to exploit NVMe's potential. Online expansion might be interesting. I tried to use ZFS for some very busy databases and ended up getting bitten badly by the fragmentation bug. The only way to restore performance appears to be copying the data off the volume, nuking it and then copying it back. Now -perhaps- if I expand the zpool then I might be able to reduce fragmentation by copying the tablespace on the same volume.

bitmagier 2y ago

Marvelous!

wkat4242 2y ago

Note: This is online expansion. Expansion was always possible but you did need to take the array down to do it. You could also move to bigger drives but you also had to do that one at a time (and only gain the new capacity once all drives were upgraded of course) As far as I know shrinking a pool is still not possible though. So if you have a pool with 5 drives and add a 6th, you can't go back to 5 drives even if there is very little data in it.

shepherdjerred 2y ago

How does ZFS compare to btrfs? I'm currently using btrfs for my home server, but I've had some strange troubles with it. I'm thinking about switching to ZFS, but I don't want to end up in the same situation.

ryao 2y ago
I first tried btrfs 15 years ago with Linux 2.6.33-rc4 if I recall. It developed an unlinkable file within 3 days, so I stopped using it. Later, I found ZFS. It had a few less significant problems, but I was a CS student at the time and I thought I could fix them since they seemed minor in comparison to the issue I had with btrfs, so over the next 18 months, I solved all of the problems that it had that bothered me and sent the patches to be included in the then ZFSOnLinux repository. My effort helped make it production ready on Linux. I have used ZFS ever since and it has worked well for me. If btrfs had been in better shape, I would have been a btrfs contributor. Unfortunately for btrfs, it not only was in bad shape back then, but other btrfs issues continued to bite me every time I tried it over the years for anything serious (e.g. frequent ENOSPC errors when there is still space). ZFS on the other hand just works. Myself and many others did a great deal of work to ensure it works well. The main reason for the difference is that ZFS had a very solid foundation, which was achieved by having some fantastic regression testing facilities. It has a userland version that randomly exercises the code to find bugs before they occur in production and a test suite that is run on every proposed change to help shake out bugs. ZFS also has more people reviewing proposed changes than other filesystems. The Btrfs developers will often state that there is a significant man power difference between the two file systems. I vaguely recall them claiming the difference was a factor of 6. Anyway, few people who use ZFS regret it, so I think you will find you like it too.
parshimers 2y ago
btrfs has similar aims to ZFS, but is far less mature. i used it for my root partitions due to it not needing DKMS, but had many troubles. i used it in a fairly simple way, just a mirror. one day, of the drives in the array started to have issues- and btrfs fell on it's face. it remounted everything read-only if i remember correctly, and would not run in degraded mode by default. even mdraid would do better than this without checksumming and so forth. ZFS also likewise, says that the array is faulted, but of course allows it to be used. the fact the default behavior was not RAID, because it's literally missing the R part for reading the data back, made me lose any faith in it. i moved to ZFS and haven't had issues since. there is much more of a community and lots of good tooling around it.