5 ms·

On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against whe

On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again.

gmueckl 2y ago
Files are always represented as lists of blocks or block spans within a file system. Individual blocks could in theory be partially shared between files at the complexity cost of a reference counter for each block. So changing a single byte in a copy on write file could take the same time regardless of file size because only the affected bock would have to be duplicated. I don't know at all how MacOS implements this copynon write scheme, though.
- MBCook 2y ago
  APFS is a copy on write filesystem if you use the right APIs, so it does what you describe but only for entire files. I believe as soon as you change a single bite you get a complete copy that’s your own. And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.
  - mintplant 2y ago
    I suppose this means that you could find yourself unexpectedly out of disk space in unintuitive ways, if you're only trying to change one byte in a cloned file but there isn't enough space to copy its entire contents?
    - pansa777 2y ago
      It doesn't work like you think. If you change one byte of duplicated file - the only "byte" will be changed on disk (a "byte", because, technically is not a byte, but a block). As far as I understand, it works like a reflink feature in the modern linux FSs. If so, thats really cool, and thats also a bit better than the zfs's snapshots. Iam newbie on macos, but it looks amazing
      - jonhohle 2y ago
        That’s true as long as the writing application only writes to blocks that have changed. Is a VM tool going to write blocks, or write multi-MB segments of a sparse image that can be swapped atomically? Unfortunately, once a file changes there are no APIs to check which blocks are still shared (at least there weren’t as of macOS 13).
    - MBCook 2y ago
      I’m not sure if it works on a file or block level for CoW, but yes. However APFS gives you a number of space related foot-guns if you want. You can overcommit partitions, for example. It also means if you have 30 GB of files on disk that could take up anywhere from a few hundred K to 30 GB of actual data depending on how many dupes you have. It’s a crazy world, but it provides some nice features.
  - tonyedgecombe 2y ago
    > I believe as soon as you change a single bite you get a complete copy that’s your own. I think it stores a delta: https://en.m.wikipedia.org/wiki/Apple_File_System#Clones https://en.m.wikipedia.org/wiki/Apple_File_System#Clones
  - alwillis 2y ago
    That’s not how this works. Nothing is deleted. It creates zero-space clones of existing files. https://en.wikipedia.org/wiki/Apple_File_System?wprov=sfti1#Clones https://en.wikipedia.org/wiki/Apple_File_System?wprov=sfti1#...
amzin 2y ago
Is there a FS that keeps only diffs in clone files? It would be neat
- rappatic 2y ago
  I wondered that too. If we only have two files, A and its duplicate B with some changes as a diff, this works pretty well. Even if the user deletes A, the OS could just apply the diff to the file on disk, unlink A, and assign B to that file. But if we have A and two different diffs B1 and B2, then try to delete A, it gets a little murkier. Either you do the above process and recalculate the diff for B2 to make it a diff of B1; or you keep the original A floating around on disk, not linked to any file. Similarly, if you try to modify A, you'd need to recalculate the diffs for all the duplicates. Alternatively, you could do version tracking and have the duplicate's diffs be on a specific version of A. Then every file would have a chain of diffs stretching back to the original content of the file. Complex but could be useful. It's certainly an interesting concept but might be more trouble than it's worth.
  - abrookewood 2y ago
    ZFS does this by de-duplicating at the block level, not the file level. It means you can do what you want without needing to keep track of a chain of differences between files. Note that de-duplication on ZFS has had issues in the past, so there is definitely a trade-off. A newer version of de-duplication sounds interesting, but I don't have any experience with it: https://www.truenas.com/docs/references/zfsdeduplication/ https://www.truenas.com/docs/references/zfsdeduplication/
- UltraSane 2y ago
  VAST storage does something like this. Unlike how most storage arrays identify the same block by hash and only store it once VAST uses a content aware hash so hashes of similar blocks are also similar. They store a reference block for each unique hash and then when new data comes in and is hashed the most similar block is used to create byte level deltas against. In practice this works extremely well. https://www.vastdata.com/blog/breaking-data-reduction-trade-offs-with-global-compression https://www.vastdata.com/blog/breaking-data-reduction-trade-...
abrookewood 2y ago
In regards to the second point, this isn't correct for ZFS: "If several files contain the same pieces (blocks) of data or any other pool data occurs more than once in the pool, ZFS stores just one copy of it. Instead of storing many copies of a book it stores one copy and an arbitrary number of pointers to that one copy." [0]. So changing one byte of a large file will not suddenly result in writing the whole file to disk again. [0] https://www.truenas.com/docs/references/zfsdeduplication/ https://www.truenas.com/docs/references/zfsdeduplication/
- karparov 2y ago
  Not the whole file but it would duplicate the block. GP didn't claim that the whole file is copied.
  - abrookewood 2y ago
    It reads like that is what they meant: "modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again"
- btilly 2y ago
  This applies to modifying a byte. But inserting a byte will change every block from then on, and will force a rewrite. Of course, that is true of most filesystems.
- taneliv 2y ago
  Yeah, I did not write it very clearly. On ZFS, you're right. On a file system that applied deduplication to files and not individual blocks, the file would need to be duplicated again, no matter where and what kind of change was made.
throwaway2037 2y ago
From a file system design perspective, does anyone know why ZFS chose to use block clones, instead of file clones?
- deleted 2y ago
  [deleted]
- raattgift 2y ago
  Records (which are of variable size) are already checksummed, and there were checksum-hashes which made it vanishingly unlikely that one could choose two different records with the same (optionally cryptographically strong) checksum-hash. When a newly-created record's checksum is generated, one could look into a table of existing checksum-hashes and avoid the record write if it already exists, substituting an incremented refcount for that table entry. ZFS is essentially an object store database at one layer; the checksum-hash deduplication table is an object like any other (file, metadata, bookmarks, ...). There is one deduplication table per pool, shared among all its datasets/volumes. On reads, one does not have to consult the dedup table. The mechanism was fairly easy to add. And for highly-deduplicatable data that is streaming-write-once-into-quiescent-pool-and-never-modify-or-delete-what's-written-into-a-deduplicated-dataset-or-volume, it was a reasonable mechanism. In other applications, the deduplication table would tend to grow and spread out, requiring extra seeks for practically every new write into a deduplicated dataset or volume, even if it's just to increment or decrement the refcount for a record. Destroying a deduplicated dataset has to decrement all its refcounts (and remove entries from the table where it's the only reference), and if your table cannot all fit in ram, the additional IOPS onto spinning media hurt, often very badly. People experimenting with deduplication and who wanted to back out after running into performance issues for typical workloads sometimes determined it was much MUCH faster to destroy the entire pool and restore from backups, rather than wait for a "zfs destroy" on a set of deduplicated snapshots/datasets/volumes to complete.
- mbreese 2y ago
  I have no specialized knowledge (just a ZFS user for over a decade). I suspect the reason is that in addition to files, ZFS will also allow you to create volumes. These volumes act like block devices, so if you want to dedup them, you need to do it at the block level.
the8472 2y ago
That's a ZFS online dedup limitation. I think xfs and btrfs are better prior art here since they use extent-based deduplication and can do offline dedup which means they don't have have to keep it in memory and the on-disk metadata is smaller too.

gmueckl 2y ago

Files are always represented as lists of blocks or block spans within a file system. Individual blocks could in theory be partially shared between files at the complexity cost of a reference counter for each block. So changing a single byte in a copy on write file could take the same time regardless of file size because only the affected bock would have to be duplicated. I don't know at all how MacOS implements this copynon write scheme, though.

MBCook 2y ago
APFS is a copy on write filesystem if you use the right APIs, so it does what you describe but only for entire files. I believe as soon as you change a single bite you get a complete copy that’s your own. And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.
- mintplant 2y ago
  I suppose this means that you could find yourself unexpectedly out of disk space in unintuitive ways, if you're only trying to change one byte in a cloned file but there isn't enough space to copy its entire contents?
  - pansa777 2y ago
    It doesn't work like you think. If you change one byte of duplicated file - the only "byte" will be changed on disk (a "byte", because, technically is not a byte, but a block). As far as I understand, it works like a reflink feature in the modern linux FSs. If so, thats really cool, and thats also a bit better than the zfs's snapshots. Iam newbie on macos, but it looks amazing
    - jonhohle 2y ago
      That’s true as long as the writing application only writes to blocks that have changed. Is a VM tool going to write blocks, or write multi-MB segments of a sparse image that can be swapped atomically? Unfortunately, once a file changes there are no APIs to check which blocks are still shared (at least there weren’t as of macOS 13).
  - MBCook 2y ago
    I’m not sure if it works on a file or block level for CoW, but yes. However APFS gives you a number of space related foot-guns if you want. You can overcommit partitions, for example. It also means if you have 30 GB of files on disk that could take up anywhere from a few hundred K to 30 GB of actual data depending on how many dupes you have. It’s a crazy world, but it provides some nice features.
- tonyedgecombe 2y ago
  > I believe as soon as you change a single bite you get a complete copy that’s your own. I think it stores a delta: https://en.m.wikipedia.org/wiki/Apple_File_System#Clones https://en.m.wikipedia.org/wiki/Apple_File_System#Clones
- alwillis 2y ago
  That’s not how this works. Nothing is deleted. It creates zero-space clones of existing files. https://en.wikipedia.org/wiki/Apple_File_System?wprov=sfti1#Clones https://en.wikipedia.org/wiki/Apple_File_System?wprov=sfti1#...

amzin 2y ago

Is there a FS that keeps only diffs in clone files? It would be neat

rappatic 2y ago
I wondered that too. If we only have two files, A and its duplicate B with some changes as a diff, this works pretty well. Even if the user deletes A, the OS could just apply the diff to the file on disk, unlink A, and assign B to that file. But if we have A and two different diffs B1 and B2, then try to delete A, it gets a little murkier. Either you do the above process and recalculate the diff for B2 to make it a diff of B1; or you keep the original A floating around on disk, not linked to any file. Similarly, if you try to modify A, you'd need to recalculate the diffs for all the duplicates. Alternatively, you could do version tracking and have the duplicate's diffs be on a specific version of A. Then every file would have a chain of diffs stretching back to the original content of the file. Complex but could be useful. It's certainly an interesting concept but might be more trouble than it's worth.
- abrookewood 2y ago
  ZFS does this by de-duplicating at the block level, not the file level. It means you can do what you want without needing to keep track of a chain of differences between files. Note that de-duplication on ZFS has had issues in the past, so there is definitely a trade-off. A newer version of de-duplication sounds interesting, but I don't have any experience with it: https://www.truenas.com/docs/references/zfsdeduplication/ https://www.truenas.com/docs/references/zfsdeduplication/
UltraSane 2y ago
VAST storage does something like this. Unlike how most storage arrays identify the same block by hash and only store it once VAST uses a content aware hash so hashes of similar blocks are also similar. They store a reference block for each unique hash and then when new data comes in and is hashed the most similar block is used to create byte level deltas against. In practice this works extremely well. https://www.vastdata.com/blog/breaking-data-reduction-trade-offs-with-global-compression https://www.vastdata.com/blog/breaking-data-reduction-trade-...

abrookewood 2y ago

In regards to the second point, this isn't correct for ZFS: "If several files contain the same pieces (blocks) of data or any other pool data occurs more than once in the pool, ZFS stores just one copy of it. Instead of storing many copies of a book it stores one copy and an arbitrary number of pointers to that one copy." [0]. So changing one byte of a large file will not suddenly result in writing the whole file to disk again. [0] https://www.truenas.com/docs/references/zfsdeduplication/ https://www.truenas.com/docs/references/zfsdeduplication/

karparov 2y ago
Not the whole file but it would duplicate the block. GP didn't claim that the whole file is copied.
- abrookewood 2y ago
  It reads like that is what they meant: "modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again"
btilly 2y ago
This applies to modifying a byte. But inserting a byte will change every block from then on, and will force a rewrite. Of course, that is true of most filesystems.
taneliv 2y ago
Yeah, I did not write it very clearly. On ZFS, you're right. On a file system that applied deduplication to files and not individual blocks, the file would need to be duplicated again, no matter where and what kind of change was made.

throwaway2037 2y ago

From a file system design perspective, does anyone know why ZFS chose to use block clones, instead of file clones?

deleted 2y ago
[deleted]
raattgift 2y ago
Records (which are of variable size) are already checksummed, and there were checksum-hashes which made it vanishingly unlikely that one could choose two different records with the same (optionally cryptographically strong) checksum-hash. When a newly-created record's checksum is generated, one could look into a table of existing checksum-hashes and avoid the record write if it already exists, substituting an incremented refcount for that table entry. ZFS is essentially an object store database at one layer; the checksum-hash deduplication table is an object like any other (file, metadata, bookmarks, ...). There is one deduplication table per pool, shared among all its datasets/volumes. On reads, one does not have to consult the dedup table. The mechanism was fairly easy to add. And for highly-deduplicatable data that is streaming-write-once-into-quiescent-pool-and-never-modify-or-delete-what's-written-into-a-deduplicated-dataset-or-volume, it was a reasonable mechanism. In other applications, the deduplication table would tend to grow and spread out, requiring extra seeks for practically every new write into a deduplicated dataset or volume, even if it's just to increment or decrement the refcount for a record. Destroying a deduplicated dataset has to decrement all its refcounts (and remove entries from the table where it's the only reference), and if your table cannot all fit in ram, the additional IOPS onto spinning media hurt, often very badly. People experimenting with deduplication and who wanted to back out after running into performance issues for typical workloads sometimes determined it was much MUCH faster to destroy the entire pool and restore from backups, rather than wait for a "zfs destroy" on a set of deduplicated snapshots/datasets/volumes to complete.
mbreese 2y ago
I have no specialized knowledge (just a ZFS user for over a decade). I suspect the reason is that in addition to files, ZFS will also allow you to create volumes. These volumes act like block devices, so if you want to dedup them, you need to do it at the block level.

the8472 2y ago

That's a ZFS online dedup limitation. I think xfs and btrfs are better prior art here since they use extent-based deduplication and can do offline dedup which means they don't have have to keep it in memory and the on-disk metadata is smaller too.