5 ms·

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writing data, I want that write to complete as fast as possible, even at the cost of disk space. That's why I don't save files I'm actively working on in 7zip archives. But later on, when the system is quiet, I would love for ZFS to go back and figure out which data is duplicated, and use the BRT or whatever to reclaim space. This could be part of a normal scrub operation.

magicalhippo 2y ago
The author of the new file-based block cloning code had this in mind. A backround process would scan files and identify dupes, delete the dupes and replace them with cloned versions. There are of course edge cases to consider to avoid data loss, but I imagine it might come soon, either officially or as a third-party tool.
Dylan16807 2y ago
The ability to alter existing snapshots, even in ways that fully preserve the data, is extremely limited in ZFS. So yes that would be great, but if I was holding my breath for Block Pointer Rewrite I'd be long dead.
- Wowfunhappy 2y ago
  You need block pointer rewrite for this?
  - Dylan16807 2y ago
    You don't need it to dedup writable files. But redundant copies in snapshots are stuck there as far as I'm aware. So if you search for duplicates every once in a while, you're not going to reap the space savings until your snapshots fully rotate.
    - lazide 2y ago
      The issue with this, in my experience, is that at some point that pro (exactly, and literally, only one copy of a specific bit of data despite many apparent copies) can become a con if there is some data corruption somewhere. Sometimes it can be a similar issue in some edge cases performance wise, but usually caching can address those problems. Efficiency being the enemy of reliability, sometimes.
      - Dylan16807 2y ago
        Redundant copies on a single volume are a waste of resources. Spend less on size, spend more on an extra parity drive, or another backup of your most important files. That way you get more safety per gigabyte.
        lazide 2y ago
        Notably, having to duplicate all data x2 (or more) is more of a waste than having 2 copies of a few files - if full drive failure is not the expected failure mode, and not all files should be protected this heavily. It’s why metadata gets duplicated in ZFS the way it does on all volumes. Having seen this play out a bunch of times, it isn’t an uncommon need either.
        Dylan16807 2y ago
        > having to duplicate all data x2 Well I didn't suggest that. I said important files only for the extra backup, and I was talking about reallocating resources not getting new ones. The simplest version is the scenario where turning on dedup means you need one less drive of space. Convert that drive to parity and you'll be better off. Split that drive from the pool and use it to backup the most important files and you'll be better off. If you can't save much space with dedup then don't bother.
        lazide 2y ago
        There was an implication in your statement that volume level was the level of granularity, yeah? I’m noting that during on volume wide dedup can have the con that you can’t choose (but it looks like you can!) to manually duplicate data.
EvanAnderson 2y ago
> I just wish we had "offline" dedupe, or even "lazy" dedupe... This is the Windows dedupe methodology. I've used it pretty extensively and I'm generally happy with it when the underlying hardware is sufficient. It's very RAM and I/O hungry but you can schedule and throttle the "groveler". I have had some data eating corruption from bugs in the Windows 2012 R2 timeframe.
UltraSane 2y ago
The neat thing about inline dedupe is that if the block hash already exists than the block doesn't have to be written. This can save a LOT of write IO in many situations. There are even extensions where a file copy between to VMs on a dedupe storage array will not actually copy any data but just increment the original blocks reference counter. You will see absurd TB/s write speeds in the OS, it is pretty cool.
- aidenn0 2y ago
  This is only a win if the dedupe table fits in RAM; otherwise you pay for it in a LOT of read IO. I have a storage array where dedupe would give me about a 2.2x reduction in disk usage, but there isn't nearly enough RAM for it.
  - UltraSane 2y ago
    yes inline dedupe has to fit in RAM. Perhaps enterprise storage arrays have spoiled me.
    - aidenn0 2y ago
      This array is a bit long-in-the-tooth and only has 192GB of RAM, but a bit over 40TB of net storage, which would be a 200GB dedup table size using the back-of-the-envelope estimate of 5GB/TB. A more precise calculation on my actual data shows that today's data would allow the dedup table to fit in RAM, but if I ever want to actually use most of the 40TB of storage, I'd need more RAM. I've had a ZFS system swap dedup to disk before, and the performance dropped to approximately zero; fixing it was a PITA, so I'm not doing that anytime soon.
      - barrkel 2y ago
        Be aware that ZFS performance rapidly drops off north of 80% utilization, when you head into 90%, you will want to buy a bigger array just to escape the pain.
hinkley 2y ago
I get the feeling that a hypothetical ZFS maintainer reading some literature on concurrent mark and sweep would be... inspirational, if not immediately helpful. You should be able to detect duplicates online. Low priority sweeping is something else. But you can at least reduce pause times.
- p_l 2y ago
  They were aware. The reasons it works the way it works are due to higher priority decisions regarding reliability in face of hardware or software corruption. That said, you can still do a two-space GC, but it's slow and possibly wasteful.
LeoPanthera 2y ago
btrfs has this. You can deduplicate a filesystem after the fact, as an overnight cron job or whatever. I really wish ZFS could do this.
- DannyBee 2y ago
  I sent a PR to add support for the necessary syscall (FIDUPERANGE) to zfs that i just have to clean up again. Once that is in, any of the existing dupe finding tools that use it (IE jdupes, duperemove) will just work on ZFS.
  - edelbitter 2y ago
    Knowing what you had to know to write that, would you dare using it? Compression, encryption and streaming sparse files together are impressive already. But now we get a new BRT entry appearing out of nowhere, dedup index pruning one that was there a moment ago, all while correctly handling arbitrary errors in whatever simultaneous deduped writes, O_DIRECT writes, FALLOC_FL_PUNCH_HOLE and reads were waiting for the same range? Sounds like adding six new places to hold the wrong lock to me.
    - DannyBee 2y ago
      "Knowing what you had to know to write that, would you dare using it?" It's no worse than anything else related to block cloning :) ZFS already supports FICLONERANGE, the thing FIDEDUPRANGE changes is that the compare is part of the atomic guarantee. So in fact, i'd argue it's actually better than what is there now - yes, the hardest part is the locking, but the locking is handled by the dedup range call getting the right locks upfront, and passing them along, so nothing else is grabbing the wrong locks. It actually has to because of the requirements to implement the ioctl properly. We have to be able to read both ranges, compare them, and clone them, all as an atomic operation wrt to concurrent writes. So instead of random things grabbing random locks, we pass the right locks around and everything verifies the locks. This means fideduprange is not as fast as it maybe could be, but it does not run into the "oops we forgot the right kind of lock" issue. At worst, it would deadlock, because it's holding exclusive locks on all that it could need before it starts to do anything in order to guarantee both the compare and the clone are atomic. So something trying to grab a lock forever under it will just deadlock. This seemed the safest course of implementation. ficlonerange is only atomic in the cloning, which means it does not have to read anything first, it can just do blind block cloning. So it actually has a more complex (but theoretically faster) lock structure because of the relaxed constraints.
DannyBee 2y ago
You can use any of the offline dupe finders to do this. Like jdupes or duperemove. I sent PR's to both the ZFS folks and the duperemove folks to support the syscalls needed. I actually have to go followup on the ZFS one, it took a while to review and i realized i completely forget to finish it up.
tiagod 2y ago
I run rdfind[1] as a cronjob to replace duplicates with hardlinks. Works fine! https://github.com/pauldreik/rdfind https://github.com/pauldreik/rdfind
- Wowfunhappy 2y ago
  But then you have to be careful not to remove the one which happens to be the "original" or the hardlinks will break, right?
  - Dylan16807 2y ago
    No, pointing to an original is how soft links work. Hard links are all equivalent. A file has any number of hard links, and at least in theory you can't distinguish between them. The risk with hardlinks is that you might alter the file. Reflinks remove that risk, and also perform very well.
    - Wowfunhappy 2y ago
      Thank you, I was unaware of this. However, the fact that editing one copy edits all of them still makes this a non-solution for me at least. I'd also strongly prefer deduping at the block level vs file level.
      - mdaniel 2y ago
        I would suspect a call to $(chmod a-w) would fix that, or at least serve as a very fine reminder that there's something special about them
- AndrewDavis 2y ago
  So this is great, if you're just looking to deduplicate read only files. Less so if you intend to write to them. Write to one and they're both updated. Anyway. Offline/lazy dedup (not in the zfs dedup sense) is something that could be done in userspace, at the file level on any filesystem that supports reflinks. When a tool like rdfind finds a duplicate, instead of replacing with a hardlink, create a copy of the file with `copy_file_range(2)` and let the filesystem create a reflink to it. Now you've got space savings and they're two separate files so if one is written to the other remains the same.
cryptonector 2y ago
Lazy/off-line dedup requires block pointer rewrite, but ZFS _cannot_ and will not ever get true BP rewrite because ZFS is not truly a CAS system. The problem is that physical locations are hashed into the Merkle hash tree, and that makes moving physical locations prohibitively expensive as you have to rewrite all the interior nodes on the way to the nodes you want to rewrite. A better design would have been to split every node that has block pointers into two sections, one that has only logical block pointers and all of whose contents gets hashed into the tree, and one that has only the physical locations (as if it were a cache) of the corresponding logical block pointers in the first section, with the second section _not_ hashed into the Merkle hash tree. Then BP rewrite would only require re-writing blocks that are not part of the Merkle hash tree. But as it is you can't get BP rewrite to work on ZFS, so you can't get what you're asking for. Well... maybe. Perhaps on read hash mismatch ZFS could attempt to locate the pointed-to block in the dedup table using the hash from the pointer. Then ZFS could reallocate the dedup'ed block. The price you'd pay then is one pointless read -- not too bad. The impossibility of BP rewrite generally leads to band-aids like this.
- Pet_Ant 2y ago
  > CAS system It looks like it means: https://en.wikipedia.org/wiki/Content-addressable_storage https://en.wikipedia.org/wiki/Content-addressable_storage
  - cryptonector 2y ago
    Sorry, yes, CAS really means that pointers are hash values -- maybe with extra metadata, yes, but _not_ including physical locations. The point is that you need some other way to map logical pointers to physical locations. The easiest way to do that is to store the mappings nearby to the references so that they are easy to find, but the mappings must be left out of the Merkle hash tree in order to make it possible to change the physical locations of the referenced blocks.
nixdev 2y ago
You can already do offline/lazy dedupe. zfs set mountpoint=foopy/foo /mnt/foo zfs set dedup=off foopy/foo zfs set mountpoint=foopy/baz /mnt/baz zfs set dedup=on foopy/baz Save all your stuff in /mnt/foo, then when you want to dedup do mv /mnt/foo/bar /mnt/baz/ Yeah... this feels like picrel, and it is https://i.pinimg.com/originals/cb/09/16/cb091697350736aae53afe4b548b9d43.jpg but it's here and now and you can do it now.

magicalhippo 2y ago

The author of the new file-based block cloning code had this in mind. A backround process would scan files and identify dupes, delete the dupes and replace them with cloned versions. There are of course edge cases to consider to avoid data loss, but I imagine it might come soon, either officially or as a third-party tool.

Dylan16807 2y ago

The ability to alter existing snapshots, even in ways that fully preserve the data, is extremely limited in ZFS. So yes that would be great, but if I was holding my breath for Block Pointer Rewrite I'd be long dead.

Wowfunhappy 2y ago
You need block pointer rewrite for this?
- Dylan16807 2y ago
  You don't need it to dedup writable files. But redundant copies in snapshots are stuck there as far as I'm aware. So if you search for duplicates every once in a while, you're not going to reap the space savings until your snapshots fully rotate.
  - lazide 2y ago
    The issue with this, in my experience, is that at some point that pro (exactly, and literally, only one copy of a specific bit of data despite many apparent copies) can become a con if there is some data corruption somewhere. Sometimes it can be a similar issue in some edge cases performance wise, but usually caching can address those problems. Efficiency being the enemy of reliability, sometimes.
    - Dylan16807 2y ago
      Redundant copies on a single volume are a waste of resources. Spend less on size, spend more on an extra parity drive, or another backup of your most important files. That way you get more safety per gigabyte.
      - lazide 2y ago
        Notably, having to duplicate all data x2 (or more) is more of a waste than having 2 copies of a few files - if full drive failure is not the expected failure mode, and not all files should be protected this heavily. It’s why metadata gets duplicated in ZFS the way it does on all volumes. Having seen this play out a bunch of times, it isn’t an uncommon need either.
        Dylan16807 2y ago
        > having to duplicate all data x2 Well I didn't suggest that. I said important files only for the extra backup, and I was talking about reallocating resources not getting new ones. The simplest version is the scenario where turning on dedup means you need one less drive of space. Convert that drive to parity and you'll be better off. Split that drive from the pool and use it to backup the most important files and you'll be better off. If you can't save much space with dedup then don't bother.
        lazide 2y ago
        There was an implication in your statement that volume level was the level of granularity, yeah? I’m noting that during on volume wide dedup can have the con that you can’t choose (but it looks like you can!) to manually duplicate data.

EvanAnderson 2y ago

> I just wish we had "offline" dedupe, or even "lazy" dedupe... This is the Windows dedupe methodology. I've used it pretty extensively and I'm generally happy with it when the underlying hardware is sufficient. It's very RAM and I/O hungry but you can schedule and throttle the "groveler". I have had some data eating corruption from bugs in the Windows 2012 R2 timeframe.

UltraSane 2y ago

The neat thing about inline dedupe is that if the block hash already exists than the block doesn't have to be written. This can save a LOT of write IO in many situations. There are even extensions where a file copy between to VMs on a dedupe storage array will not actually copy any data but just increment the original blocks reference counter. You will see absurd TB/s write speeds in the OS, it is pretty cool.

aidenn0 2y ago
This is only a win if the dedupe table fits in RAM; otherwise you pay for it in a LOT of read IO. I have a storage array where dedupe would give me about a 2.2x reduction in disk usage, but there isn't nearly enough RAM for it.
- UltraSane 2y ago
  yes inline dedupe has to fit in RAM. Perhaps enterprise storage arrays have spoiled me.
  - aidenn0 2y ago
    This array is a bit long-in-the-tooth and only has 192GB of RAM, but a bit over 40TB of net storage, which would be a 200GB dedup table size using the back-of-the-envelope estimate of 5GB/TB. A more precise calculation on my actual data shows that today's data would allow the dedup table to fit in RAM, but if I ever want to actually use most of the 40TB of storage, I'd need more RAM. I've had a ZFS system swap dedup to disk before, and the performance dropped to approximately zero; fixing it was a PITA, so I'm not doing that anytime soon.
    - barrkel 2y ago
      Be aware that ZFS performance rapidly drops off north of 80% utilization, when you head into 90%, you will want to buy a bigger array just to escape the pain.

hinkley 2y ago

I get the feeling that a hypothetical ZFS maintainer reading some literature on concurrent mark and sweep would be... inspirational, if not immediately helpful. You should be able to detect duplicates online. Low priority sweeping is something else. But you can at least reduce pause times.

p_l 2y ago
They were aware. The reasons it works the way it works are due to higher priority decisions regarding reliability in face of hardware or software corruption. That said, you can still do a two-space GC, but it's slow and possibly wasteful.

LeoPanthera 2y ago

btrfs has this. You can deduplicate a filesystem after the fact, as an overnight cron job or whatever. I really wish ZFS could do this.

DannyBee 2y ago
I sent a PR to add support for the necessary syscall (FIDUPERANGE) to zfs that i just have to clean up again. Once that is in, any of the existing dupe finding tools that use it (IE jdupes, duperemove) will just work on ZFS.
- edelbitter 2y ago
  Knowing what you had to know to write that, would you dare using it? Compression, encryption and streaming sparse files together are impressive already. But now we get a new BRT entry appearing out of nowhere, dedup index pruning one that was there a moment ago, all while correctly handling arbitrary errors in whatever simultaneous deduped writes, O_DIRECT writes, FALLOC_FL_PUNCH_HOLE and reads were waiting for the same range? Sounds like adding six new places to hold the wrong lock to me.
  - DannyBee 2y ago
    "Knowing what you had to know to write that, would you dare using it?" It's no worse than anything else related to block cloning :) ZFS already supports FICLONERANGE, the thing FIDEDUPRANGE changes is that the compare is part of the atomic guarantee. So in fact, i'd argue it's actually better than what is there now - yes, the hardest part is the locking, but the locking is handled by the dedup range call getting the right locks upfront, and passing them along, so nothing else is grabbing the wrong locks. It actually has to because of the requirements to implement the ioctl properly. We have to be able to read both ranges, compare them, and clone them, all as an atomic operation wrt to concurrent writes. So instead of random things grabbing random locks, we pass the right locks around and everything verifies the locks. This means fideduprange is not as fast as it maybe could be, but it does not run into the "oops we forgot the right kind of lock" issue. At worst, it would deadlock, because it's holding exclusive locks on all that it could need before it starts to do anything in order to guarantee both the compare and the clone are atomic. So something trying to grab a lock forever under it will just deadlock. This seemed the safest course of implementation. ficlonerange is only atomic in the cloning, which means it does not have to read anything first, it can just do blind block cloning. So it actually has a more complex (but theoretically faster) lock structure because of the relaxed constraints.

DannyBee 2y ago

You can use any of the offline dupe finders to do this. Like jdupes or duperemove. I sent PR's to both the ZFS folks and the duperemove folks to support the syscalls needed. I actually have to go followup on the ZFS one, it took a while to review and i realized i completely forget to finish it up.

tiagod 2y ago

I run rdfind[1] as a cronjob to replace duplicates with hardlinks. Works fine! https://github.com/pauldreik/rdfind https://github.com/pauldreik/rdfind

Wowfunhappy 2y ago
But then you have to be careful not to remove the one which happens to be the "original" or the hardlinks will break, right?
- Dylan16807 2y ago
  No, pointing to an original is how soft links work. Hard links are all equivalent. A file has any number of hard links, and at least in theory you can't distinguish between them. The risk with hardlinks is that you might alter the file. Reflinks remove that risk, and also perform very well.
  - Wowfunhappy 2y ago
    Thank you, I was unaware of this. However, the fact that editing one copy edits all of them still makes this a non-solution for me at least. I'd also strongly prefer deduping at the block level vs file level.
    - mdaniel 2y ago
      I would suspect a call to $(chmod a-w) would fix that, or at least serve as a very fine reminder that there's something special about them
AndrewDavis 2y ago
So this is great, if you're just looking to deduplicate read only files. Less so if you intend to write to them. Write to one and they're both updated. Anyway. Offline/lazy dedup (not in the zfs dedup sense) is something that could be done in userspace, at the file level on any filesystem that supports reflinks. When a tool like rdfind finds a duplicate, instead of replacing with a hardlink, create a copy of the file with `copy_file_range(2)` and let the filesystem create a reflink to it. Now you've got space savings and they're two separate files so if one is written to the other remains the same.

cryptonector 2y ago

Lazy/off-line dedup requires block pointer rewrite, but ZFS _cannot_ and will not ever get true BP rewrite because ZFS is not truly a CAS system. The problem is that physical locations are hashed into the Merkle hash tree, and that makes moving physical locations prohibitively expensive as you have to rewrite all the interior nodes on the way to the nodes you want to rewrite. A better design would have been to split every node that has block pointers into two sections, one that has only logical block pointers and all of whose contents gets hashed into the tree, and one that has only the physical locations (as if it were a cache) of the corresponding logical block pointers in the first section, with the second section _not_ hashed into the Merkle hash tree. Then BP rewrite would only require re-writing blocks that are not part of the Merkle hash tree. But as it is you can't get BP rewrite to work on ZFS, so you can't get what you're asking for. Well... maybe. Perhaps on read hash mismatch ZFS could attempt to locate the pointed-to block in the dedup table using the hash from the pointer. Then ZFS could reallocate the dedup'ed block. The price you'd pay then is one pointless read -- not too bad. The impossibility of BP rewrite generally leads to band-aids like this.

Pet_Ant 2y ago
> CAS system It looks like it means: https://en.wikipedia.org/wiki/Content-addressable_storage https://en.wikipedia.org/wiki/Content-addressable_storage
- cryptonector 2y ago
  Sorry, yes, CAS really means that pointers are hash values -- maybe with extra metadata, yes, but _not_ including physical locations. The point is that you need some other way to map logical pointers to physical locations. The easiest way to do that is to store the mappings nearby to the references so that they are easy to find, but the mappings must be left out of the Merkle hash tree in order to make it possible to change the physical locations of the referenced blocks.

nixdev 2y ago

You can already do offline/lazy dedupe. zfs set mountpoint=foopy/foo /mnt/foo zfs set dedup=off foopy/foo zfs set mountpoint=foopy/baz /mnt/baz zfs set dedup=on foopy/baz Save all your stuff in /mnt/foo, then when you want to dedup do mv /mnt/foo/bar /mnt/baz/ Yeah... this feels like picrel, and it is https://i.pinimg.com/originals/cb/09/16/cb091697350736aae53afe4b548b9d43.jpg but it's here and now and you can do it now.