5 ms·

I've always wondered why file systems don't have a dedupe feature similar to defragment. Imagine an inode filled with all zeroes for example, tons of files have

by _8j50 3y ago

I've always wondered why file systems don't have a dedupe feature similar to defragment. Imagine an inode filled with all zeroes for example, tons of files have large segments that are all zeroes, what if they all get deduped and point to the same inode when the identical bytes are >= the inode size? There must be inode caching speedups for this I'd think. Saved disk space is also an advantage, especially if you work with large uncompressed files a lot (e.g. media editing/archiving).

TheDong 3y ago
zfs has "zfs set dedupe=on", which does exactly what you would think. There's also compression, and compression ends up deduplicating data fairly naturally. Other filesystems with compression (like bcachefs) also naturally manage some level of deduplication through compression.
- pseudalopex 3y ago
  Compression is not what anyone means when they talk about deduplicating file systems. Or what the 1st comment described.
  - linsomniac 3y ago
    Agreed, but I think they were meaning "compression gets rid of long strings of NULs".
wongarsu 3y ago
Both ZFS and Btrfs can do this. ZFS does it live, checking the hash of any block that is to be written against the hashes of all existing blocks. That requires keeping those hashes in memory, costing you a non-trivial amount of RAM if you turn it on. It also forces a slower block hash than ZFS would normally choose because collisions matter more. It can be great for read speeds and storage, but write speeds suffer. Btrfs instead writes normally, but has a command you can run to deduplicate existing blocks. There is also an option to do this at a file level instead of the block level, which decreases the overhead but leads to fewer storage gains.
- spandextwins 3y ago
  Absolutely. Our experience with zfs dedupe was that it was ok for some workloads, like remote desktop stuff, but not for most, not worth the overhead. ZFS compression, on the other hand, was well worth the extra CPU time. But fdupes is great for finding files exactly the same, great tool!
mustache_kimono 3y ago
> Imagine an inode filled with all zeroes for example, tons of files have large segments that are all zeroes, what if they all get deduped and point to the same inode when the identical bytes are >= the inode size? Transparent compression of ZFS handles trivial cases like this quite easily. It even has a 'zle' (Zero Length Encoding) compression mode expressly for such, blocks with all zeros only, instances. Although, of course, the default 'lz4' compression should cover such trivial instances equally well. > I've always wondered why file systems don't have a dedupe feature similar to defragment. Because dedup is enormously expensive to do in the otherwise non-trivial cases. ZFS can do it live, and it requires an in memory hash table to track each deduped block. Which, if you imagine a ZFS filesystem with a recordsize of 8K or 16K blocks, this can be enormously RAM intensive. btrfs supposedly does dedup with less overhead(?), but I'd be interested to hear about instance where the combination of compression and filesystem clones isn't more practical.
- _8j50 3y ago
  Thanks, TIL!
doctorpangloss 3y ago
Windows does, for both NTFS and ReFS. It works flawlessly in my experience. You can enable deduplication with one PowerShell command. I’m sure Microsoft would enable it for Windows 11, instead of just Windows 2022, if it didn’t eat directly into juicy margins on increased storage in an industry where product customization / personalization is like 10% of revenues but 40% of profits. That one step would probably free up global exabytes of disk space.
- wrs 3y ago
  Just to be precise, it’s not part of the filesystem. It’s implemented on top of the filesystem as a background process that moves duplicate data into a “chunk store” and replaces the files with reparse points that route accesses to the deduplication driver. This is ostensibly to keep deduplication out of the write path for performance reasons. https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/understand https://learn.microsoft.com/en-us/windows-server/storage/dat...
dreamcompiler 3y ago
Recent versions of MacOS with AFS can do this to some extent: At the time you make a copy of a file with the Finder, MacOS makes a COW (copy-on-write) copy of the file which behaves like a hardlink until you modify one of the copies, at which point the modified one becomes a separate file with its own identity. But AFAIK MacOS doesn't attempt to proactively dedupe the file system.

by _8j50 3y ago

TheDong 3y ago

zfs has "zfs set dedupe=on", which does exactly what you would think. There's also compression, and compression ends up deduplicating data fairly naturally. Other filesystems with compression (like bcachefs) also naturally manage some level of deduplication through compression.

pseudalopex 3y ago
Compression is not what anyone means when they talk about deduplicating file systems. Or what the 1st comment described.
- linsomniac 3y ago
  Agreed, but I think they were meaning "compression gets rid of long strings of NULs".

wongarsu 3y ago

Both ZFS and Btrfs can do this. ZFS does it live, checking the hash of any block that is to be written against the hashes of all existing blocks. That requires keeping those hashes in memory, costing you a non-trivial amount of RAM if you turn it on. It also forces a slower block hash than ZFS would normally choose because collisions matter more. It can be great for read speeds and storage, but write speeds suffer. Btrfs instead writes normally, but has a command you can run to deduplicate existing blocks. There is also an option to do this at a file level instead of the block level, which decreases the overhead but leads to fewer storage gains.

spandextwins 3y ago
Absolutely. Our experience with zfs dedupe was that it was ok for some workloads, like remote desktop stuff, but not for most, not worth the overhead. ZFS compression, on the other hand, was well worth the extra CPU time. But fdupes is great for finding files exactly the same, great tool!

mustache_kimono 3y ago

> Imagine an inode filled with all zeroes for example, tons of files have large segments that are all zeroes, what if they all get deduped and point to the same inode when the identical bytes are >= the inode size? Transparent compression of ZFS handles trivial cases like this quite easily. It even has a 'zle' (Zero Length Encoding) compression mode expressly for such, blocks with all zeros only, instances. Although, of course, the default 'lz4' compression should cover such trivial instances equally well. > I've always wondered why file systems don't have a dedupe feature similar to defragment. Because dedup is enormously expensive to do in the otherwise non-trivial cases. ZFS can do it live, and it requires an in memory hash table to track each deduped block. Which, if you imagine a ZFS filesystem with a recordsize of 8K or 16K blocks, this can be enormously RAM intensive. btrfs supposedly does dedup with less overhead(?), but I'd be interested to hear about instance where the combination of compression and filesystem clones isn't more practical.

_8j50 3y ago
Thanks, TIL!

doctorpangloss 3y ago

Windows does, for both NTFS and ReFS. It works flawlessly in my experience. You can enable deduplication with one PowerShell command. I’m sure Microsoft would enable it for Windows 11, instead of just Windows 2022, if it didn’t eat directly into juicy margins on increased storage in an industry where product customization / personalization is like 10% of revenues but 40% of profits. That one step would probably free up global exabytes of disk space.

wrs 3y ago
Just to be precise, it’s not part of the filesystem. It’s implemented on top of the filesystem as a background process that moves duplicate data into a “chunk store” and replaces the files with reparse points that route accesses to the deduplication driver. This is ostensibly to keep deduplication out of the write path for performance reasons. https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/understand https://learn.microsoft.com/en-us/windows-server/storage/dat...

dreamcompiler 3y ago

Recent versions of MacOS with AFS can do this to some extent: At the time you make a copy of a file with the Finder, MacOS makes a COW (copy-on-write) copy of the file which behaves like a hardlink until you modify one of the copies, at which point the modified one becomes a separate file with its own identity. But AFAIK MacOS doesn't attempt to proactively dedupe the file system.