3 ms·
Both ZFS and Btrfs can do this. ZFS does it live, checking the hash of any block that is to be written against the hashes of all existing blocks. That requires
by wongarsu 3y ago
Both ZFS and Btrfs can do this.
ZFS does it live, checking the hash of any block that is to be written against the hashes of all existing blocks. That requires keeping those hashes in memory, costing you a non-trivial amount of RAM if you turn it on. It also forces a slower block hash than ZFS would normally choose because collisions matter more. It can be great for read speeds and storage, but write speeds suffer.
Btrfs instead writes normally, but has a command you can run to deduplicate existing blocks. There is also an option to do this at a file level instead of the block level, which decreases the overhead but leads to fewer storage gains.
- spandextwins 3y agoAbsolutely. Our experience with zfs dedupe was that it was ok for some workloads, like remote desktop stuff, but not for most, not worth the overhead. ZFS compression, on the other hand, was well worth the extra CPU time. But fdupes is great for finding files exactly the same, great tool!