3 ms·
That only seems to work on btrfs, XFS, (and maybe now or very soon) ZFS and bcachefs: "[duperemove] simply finds candidates for dedupe and submits them to the L
by zeotroph 2y ago
That only seems to work on btrfs, XFS, (and maybe now or very soon) ZFS and bcachefs: "[duperemove] simply finds candidates for dedupe and submits them to the Linux kernel FIDEDUPERANGE ioctl." [1] (aka BTRFS_IOC_FILE_EXTENT_SAME), and this ioctl "performs the 'compare and share if identical'" (and locking etc.) work [2]. But on those filesystems, that is a nice feature, plus it lets the tool get away with a weak hash like murmur3.
1: http://markfasheh.github.io/duperemove/duperemove.html http://markfasheh.github.io/duperemove/duperemove.html
2: https://manpages.debian.org/bookworm/manpages-dev/ioctl_fideduperange.2.en.html https://manpages.debian.org/bookworm/manpages-dev/ioctl_fide...
- Arnavion 2y agoA fallback to hardlinking would work for a lot more filesystems, though of course it would be limited to files that are completely identical rather than any identical extents / blocks within files.
- gavinsyancey 2y agoMore importantly, hardlinking means writes to one file would affect both, while FIDUPERANGE creates copy-on-write semantics.
- JackSlateur 2y agoQuick note : I removed murmur3 support, and replaced it with xxhash