4 ms·

After years in the making ZFS raidz expansaion is finally here. Major features added in release: - RAIDZ Expansion: Add new devices to an existing RAIDZ poo

by scrp 2y ago

After years in the making ZFS raidz expansaion is finally here. Major features added in release: - RAIDZ Expansion: Add new devices to an existing RAIDZ pool, increasing storage capacity without downtime. - Fast Dedup: A major performance upgrade to the original OpenZFS deduplication functionality. - Direct IO: Allows bypassing the ARC for reads/writes, improving performance in scenarios like NVMe devices where caching may hinder efficiency. - JSON: Optional JSON output for the most used commands. - Long names: Support for file and directory names up to 1023 characters.

cm2187 2y ago
But I presume it is still not possible to remove a vdev.
- mustache_kimono 2y ago
  Is this possible elsewhere (re: other filesystems)?
  - cm2187 2y ago
    It is possible with windows storage space (remove drive from a pool) and mdadm/lvm (remove disk from a RAID array, remove volume from lvm), which to me are the two major alternatives. Don't know about unraid.
    - mustache_kimono 2y ago
      > It is possible with windows storage space (remove drive from a pool) and mdadm/lvm (remove disk from a RAID array, remove volume from lvm), which to me are the two major alternatives. Don't know about unraid. Perhaps I am misunderstanding you, but you can offline and remove drives from a ZFS pool. Do you mean WSS and mdadm/lvm will allow an automatic live rebalance and then reconfigure the drive topology?
      - cm2187 2y ago
        So for instance I have a ZFS pool with 3 HDD data vdevs, and 2 SSD special vdevs. I want to convert the two SSD vdevs into a single one (or possibly remove one of them). From what I read the only way to do that is to destroy the entire pool and recreate it (it's in a server in a datacentre, don't want to reupload that much data). In windows, you can set a disk for removal, and as long as the other disks have enough space and are compatible with the virtual disks (eg you need at least 5 disks if you have parity with number of columns=5), it will rebalance the blocks onto the other disks until you can safely remove the disk. If you use thin provisioning, you can also change your mind about the settings of a virtual disk, create a new one on the same pool, and move the data from one to the other. Mdadm/lvm will do the same albeit with more of a pain in the arse as RAID requires to resilver not just the occupied space but also the free space so takes a lot more time and IO than it should. It's one of my beef with ZFS, there are lots of no return decisions. That and I ran into some race conditions with loading a ZFS array on boot with nvme drives on ubuntu. They seem to not be ready, resulting in randomly degraded arrays. Fixed by loading the pool with a delay.
        ryao 2y ago
        The man page says that your example is doable with zpool remove: https://openzfs.github.io/openzfs-docs/man/master/8/zpool-remove.8.html https://openzfs.github.io/openzfs-docs/man/master/8/zpool-re...
        formerly_proven 2y ago
        My understanding is that ZFS does virtual <-> physical translation in the vdev layer, i.e. all block references in ZFS contain a (vdev, vblock) tuple, and the vdev knows how to translate that virtual block offset into actual on-disk block offset(s). This kinda implies that you can't actually remove data vdevs, because in practice you can't rewrite all references. You also can't do offline deduplication without rewriting references (i.e. actually touching the files in the filesystem). And that's why ZFS can't deduplicate snapshots after the fact. On the other hand, reshaping a vdev is possible, because that "just" requires shuffling the vblock -> physical block associations inside the vdev.
    - lloeki 2y ago
      IIUC the ask (I have a hard time wrapping my head around zfs vernacular), btrfs allows this at least in some cases. If you can convince btrfs balance to not use the dev to remove it will simply rebalance data to the other devs and then you can btrfs device remove.
  - c45y 2y ago
    Bcachefs allows it
    - eptcyka 2y ago
      Cool, just have to wait before it is stable enough for daily use of mission critical data. I am personally optimistic about bcachefs, but incredibly pessimistic about changing filesystems.
      - ryao 2y ago
        It seems easier to copy data to a new ZFS pool if you need to remove RAID-Z top level vdevs. Another possibility is to just wait for someone to implement it in ZFS. ZFS already has top level vdev removal for other types of vdevs. Support for top level raid-z vdev removal just needs to be implemented on top of that.
  - unixhero 2y ago
    Btrfs
    - tw04 2y ago
      Except you shouldn’t use btrfs for any parity based raid if you value your data at all. In fact, I’m not aware if any vendor that has implemented btrfs with parity based raid, they all resort to btrfs on md.
  - pantalaimon 2y ago
    btrfs has supported online adding and removing of devices to the pool from the start
    - allanjude 2y ago
      ZFS has supported online adding of vdevs since the start too, this is specifically modifying an existing vdev and widening it, which is much less common, and much more complex
- ryao 2y ago
  That was added a while ago: https://openzfs.github.io/openzfs-docs/man/master/8/zpool-remove.8.html https://openzfs.github.io/openzfs-docs/man/master/8/zpool-re... It works by making a readonly copy of the vdev being removed inside the remaining space. The existing vdev is then removed. Data can still be accessed from the copy, but new writes will go to an actual vdev while data no longer needed on the copy is gradually reclaimed as free space as the old data is no longer needed.
  - lutorm 2y ago
    Although "Top-level vdevs can only be removed if the primary pool storage does not contain a top-level raidz vdev, all top-level vdevs have the same sector size, and the keys for all encrypted datasets are loaded."
    - ryao 2y ago
      I forgot we still did not have that last bit implemented. However, it is less important now that we have expansion.
      - justinclift 2y ago
        > However, it is less important now that we have expansion. Not really sure if that's true. They seem like two different/distinct use cases, though there's probably some small overlap.
    - cm2187 2y ago
      And in my case all the vdevs are raidz
jdboyd 2y ago
The first 4 seem like really big deals.
- snvzz 2y ago
  The fifth is also, once you consider non-ascii names.
  - GeorgeTirebiter 2y ago
    Could someone show a legit reason to use 1000-character filenames? Seems to me, when filenames are long like that, they are actually capturing several KEYS that can be easily searched via ls & re's. e.g. 2025-Jan-14-1258.93743_Experiment-2345_Gas-Flow-375.3_etc_etc.dat But to me this stuff should be in metadata. It's just that we don't have great tools for grepping the metadata. Heck, the original Macintosh FS had no subdirectories - they were faked by burying subdirectory names in the (flat filesysytem) filename. The original Macintosh File System (MFS), did not support true hierarchical subdirectories. Instead, the illusion of subdirectories was created by embedding folder-like names into the filenames themselves. This was done by using colons (:) as separators in filenames. A file named Folder:Subfolder:File would appear to belong to a subfolder within a folder. This was entirely a user interface convention managed by the Finder. Internally, MFS stored all files in a flat namespace, with no actual directory hierarchy in the filesystem structure. So, there is 'utility' in "overloading the filename space". But...
    - p_l 2y ago
      > Could someone show a legit reason to use 1000-character filenames? 1023 byte names can mean less than 250 characters due to use of unicode and utf-8. Add to it unicode normalization which might "expand" some characters into two or more combining characters, deliberate use of combining characters, emoji, rare characters, and you might end up with many "characters" taking more than 4 bytes. A single "country flag" character will be usually 8 bytes, usually most emoji will be at least 4 bytes, skin tone modifiers will add 4 bytes, etc. this ' ' takes 27 bytes in my terminal, '󠁧󠁢󠁳󠁣󠁴󠁿' takes 28, another combo I found is 35 bytes. And that's on top of just getting a long title using let's say one of CJK or other less common scripts - an early manuscript of somewhat successful Japanese novel has a non-normalized filename of 119 byte, and it's nowhere close to actually long titles, something that someone might reasonably have on disk. A random find on the internet easily points to a book title that takes over 300 bytes in non-normalized utf8. P.S. proper title of "Robinson Crusoe" if used as filename takes at least 395 bytes...
eatbitseveryday 2y ago
> RAIDZ Expansion: Add new devices to an existing RAIDZ pool, increasing storage capacity without downtime. More specifically: > A new device (disk) can be attached to an existing RAIDZ vdev
BodyCulture 2y ago
How well tested is this in combination with encryption? Is the ZFS team handling encryption as a first class priority at all? ZFS on Linux inherited a lot of fame from ZFS on Solaris, but everyone using it in production should study the issue tracker very well for a realistic impression of the situation.
- p_l 2y ago
  Main issue with encryption is occasional attempts by certain (specific) Linux kernel developer to lockout ZFS out of access to advanced instruction set extensions (far from the only weird idea of that specific developer). The way ZFS encryption is layered, the features should be pretty much orthogonal from each other, but I'll admit that there's a bit of lacking with ZFS native encryption (though mainly in upper layer tooling in my experience rather than actual on-disk encryption parts)
  - snvzz 2y ago
    >occasional attempts by certain (specific) Linux kernel developer Can we please refer to them by the actual name?
    - E39M5S62 2y ago
      Greg Kroah-Hartman.
  - ryao 2y ago
    These are actually wrappers around CPU instructions, so what ZFS does is implement its own equivalents. This does not affect encryption (beyond the inconvenience that we did not have SIMD acceleration for a while on certain architectures).
- deleted 2y ago
  [deleted]
- ryao 2y ago
  The new features should interact fine with encryption. They are implemented at different parts of ZFS' internal stack. There have been many man hours put into investigating bug reports involving encryption and some fixes were made. Unfortunately, something appears to be going wrong when non-raw sends of encrypted datasets are received by another system: https://github.com/openzfs/zfs/issues/12014 https://github.com/openzfs/zfs/issues/12014 I do not believe anyone has figured out what is going wrong there. It has not been for lack of trying. Raw sends from encrypted datasets appear to be fine.
cromka 2y ago
So if I’m running a Proxmox on ZFS and NVMEs, will I be better off enabling Direct IO when 2.3 gets rolled out? What are the use cases for it?
- Saris 2y ago
  I would guess for very high performance NVMe drives.
- 0x457 2y ago
  Direct IO useful for databases and other applications that use their own disk caching layer. Without knowing what you run in Proxmox no one will be able to tell you if it's beneficial or not.

by scrp 2y ago

cm2187 2y ago

But I presume it is still not possible to remove a vdev.

mustache_kimono 2y ago
Is this possible elsewhere (re: other filesystems)?
- cm2187 2y ago
  It is possible with windows storage space (remove drive from a pool) and mdadm/lvm (remove disk from a RAID array, remove volume from lvm), which to me are the two major alternatives. Don't know about unraid.
  - mustache_kimono 2y ago
    > It is possible with windows storage space (remove drive from a pool) and mdadm/lvm (remove disk from a RAID array, remove volume from lvm), which to me are the two major alternatives. Don't know about unraid. Perhaps I am misunderstanding you, but you can offline and remove drives from a ZFS pool. Do you mean WSS and mdadm/lvm will allow an automatic live rebalance and then reconfigure the drive topology?
    - cm2187 2y ago
      So for instance I have a ZFS pool with 3 HDD data vdevs, and 2 SSD special vdevs. I want to convert the two SSD vdevs into a single one (or possibly remove one of them). From what I read the only way to do that is to destroy the entire pool and recreate it (it's in a server in a datacentre, don't want to reupload that much data). In windows, you can set a disk for removal, and as long as the other disks have enough space and are compatible with the virtual disks (eg you need at least 5 disks if you have parity with number of columns=5), it will rebalance the blocks onto the other disks until you can safely remove the disk. If you use thin provisioning, you can also change your mind about the settings of a virtual disk, create a new one on the same pool, and move the data from one to the other. Mdadm/lvm will do the same albeit with more of a pain in the arse as RAID requires to resilver not just the occupied space but also the free space so takes a lot more time and IO than it should. It's one of my beef with ZFS, there are lots of no return decisions. That and I ran into some race conditions with loading a ZFS array on boot with nvme drives on ubuntu. They seem to not be ready, resulting in randomly degraded arrays. Fixed by loading the pool with a delay.
      - ryao 2y ago
        The man page says that your example is doable with zpool remove: https://openzfs.github.io/openzfs-docs/man/master/8/zpool-remove.8.html https://openzfs.github.io/openzfs-docs/man/master/8/zpool-re...
      - formerly_proven 2y ago
        My understanding is that ZFS does virtual <-> physical translation in the vdev layer, i.e. all block references in ZFS contain a (vdev, vblock) tuple, and the vdev knows how to translate that virtual block offset into actual on-disk block offset(s). This kinda implies that you can't actually remove data vdevs, because in practice you can't rewrite all references. You also can't do offline deduplication without rewriting references (i.e. actually touching the files in the filesystem). And that's why ZFS can't deduplicate snapshots after the fact. On the other hand, reshaping a vdev is possible, because that "just" requires shuffling the vblock -> physical block associations inside the vdev.
  - lloeki 2y ago
    IIUC the ask (I have a hard time wrapping my head around zfs vernacular), btrfs allows this at least in some cases. If you can convince btrfs balance to not use the dev to remove it will simply rebalance data to the other devs and then you can btrfs device remove.
- c45y 2y ago
  Bcachefs allows it
  - eptcyka 2y ago
    Cool, just have to wait before it is stable enough for daily use of mission critical data. I am personally optimistic about bcachefs, but incredibly pessimistic about changing filesystems.
    - ryao 2y ago
      It seems easier to copy data to a new ZFS pool if you need to remove RAID-Z top level vdevs. Another possibility is to just wait for someone to implement it in ZFS. ZFS already has top level vdev removal for other types of vdevs. Support for top level raid-z vdev removal just needs to be implemented on top of that.
- unixhero 2y ago
  Btrfs
  - tw04 2y ago
    Except you shouldn’t use btrfs for any parity based raid if you value your data at all. In fact, I’m not aware if any vendor that has implemented btrfs with parity based raid, they all resort to btrfs on md.
- pantalaimon 2y ago
  btrfs has supported online adding and removing of devices to the pool from the start
  - allanjude 2y ago
    ZFS has supported online adding of vdevs since the start too, this is specifically modifying an existing vdev and widening it, which is much less common, and much more complex
ryao 2y ago
That was added a while ago: https://openzfs.github.io/openzfs-docs/man/master/8/zpool-remove.8.html https://openzfs.github.io/openzfs-docs/man/master/8/zpool-re... It works by making a readonly copy of the vdev being removed inside the remaining space. The existing vdev is then removed. Data can still be accessed from the copy, but new writes will go to an actual vdev while data no longer needed on the copy is gradually reclaimed as free space as the old data is no longer needed.
- lutorm 2y ago
  Although "Top-level vdevs can only be removed if the primary pool storage does not contain a top-level raidz vdev, all top-level vdevs have the same sector size, and the keys for all encrypted datasets are loaded."
  - ryao 2y ago
    I forgot we still did not have that last bit implemented. However, it is less important now that we have expansion.
    - justinclift 2y ago
      > However, it is less important now that we have expansion. Not really sure if that's true. They seem like two different/distinct use cases, though there's probably some small overlap.
  - cm2187 2y ago
    And in my case all the vdevs are raidz

jdboyd 2y ago

The first 4 seem like really big deals.

snvzz 2y ago
The fifth is also, once you consider non-ascii names.
- GeorgeTirebiter 2y ago
  Could someone show a legit reason to use 1000-character filenames? Seems to me, when filenames are long like that, they are actually capturing several KEYS that can be easily searched via ls & re's. e.g. 2025-Jan-14-1258.93743_Experiment-2345_Gas-Flow-375.3_etc_etc.dat But to me this stuff should be in metadata. It's just that we don't have great tools for grepping the metadata. Heck, the original Macintosh FS had no subdirectories - they were faked by burying subdirectory names in the (flat filesysytem) filename. The original Macintosh File System (MFS), did not support true hierarchical subdirectories. Instead, the illusion of subdirectories was created by embedding folder-like names into the filenames themselves. This was done by using colons (:) as separators in filenames. A file named Folder:Subfolder:File would appear to belong to a subfolder within a folder. This was entirely a user interface convention managed by the Finder. Internally, MFS stored all files in a flat namespace, with no actual directory hierarchy in the filesystem structure. So, there is 'utility' in "overloading the filename space". But...
  - p_l 2y ago
    > Could someone show a legit reason to use 1000-character filenames? 1023 byte names can mean less than 250 characters due to use of unicode and utf-8. Add to it unicode normalization which might "expand" some characters into two or more combining characters, deliberate use of combining characters, emoji, rare characters, and you might end up with many "characters" taking more than 4 bytes. A single "country flag" character will be usually 8 bytes, usually most emoji will be at least 4 bytes, skin tone modifiers will add 4 bytes, etc. this ' ' takes 27 bytes in my terminal, '󠁧󠁢󠁳󠁣󠁴󠁿' takes 28, another combo I found is 35 bytes. And that's on top of just getting a long title using let's say one of CJK or other less common scripts - an early manuscript of somewhat successful Japanese novel has a non-normalized filename of 119 byte, and it's nowhere close to actually long titles, something that someone might reasonably have on disk. A random find on the internet easily points to a book title that takes over 300 bytes in non-normalized utf8. P.S. proper title of "Robinson Crusoe" if used as filename takes at least 395 bytes...

eatbitseveryday 2y ago

> RAIDZ Expansion: Add new devices to an existing RAIDZ pool, increasing storage capacity without downtime. More specifically: > A new device (disk) can be attached to an existing RAIDZ vdev

BodyCulture 2y ago

How well tested is this in combination with encryption? Is the ZFS team handling encryption as a first class priority at all? ZFS on Linux inherited a lot of fame from ZFS on Solaris, but everyone using it in production should study the issue tracker very well for a realistic impression of the situation.

p_l 2y ago
Main issue with encryption is occasional attempts by certain (specific) Linux kernel developer to lockout ZFS out of access to advanced instruction set extensions (far from the only weird idea of that specific developer). The way ZFS encryption is layered, the features should be pretty much orthogonal from each other, but I'll admit that there's a bit of lacking with ZFS native encryption (though mainly in upper layer tooling in my experience rather than actual on-disk encryption parts)
- snvzz 2y ago
  >occasional attempts by certain (specific) Linux kernel developer Can we please refer to them by the actual name?
  - E39M5S62 2y ago
    Greg Kroah-Hartman.
- ryao 2y ago
  These are actually wrappers around CPU instructions, so what ZFS does is implement its own equivalents. This does not affect encryption (beyond the inconvenience that we did not have SIMD acceleration for a while on certain architectures).
deleted 2y ago
[deleted]
ryao 2y ago
The new features should interact fine with encryption. They are implemented at different parts of ZFS' internal stack. There have been many man hours put into investigating bug reports involving encryption and some fixes were made. Unfortunately, something appears to be going wrong when non-raw sends of encrypted datasets are received by another system: https://github.com/openzfs/zfs/issues/12014 https://github.com/openzfs/zfs/issues/12014 I do not believe anyone has figured out what is going wrong there. It has not been for lack of trying. Raw sends from encrypted datasets appear to be fine.

cromka 2y ago

So if I’m running a Proxmox on ZFS and NVMEs, will I be better off enabling Direct IO when 2.3 gets rolled out? What are the use cases for it?

Saris 2y ago
I would guess for very high performance NVMe drives.
0x457 2y ago
Direct IO useful for databases and other applications that use their own disk caching layer. Without knowing what you run in Proxmox no one will be able to tell you if it's beneficial or not.