4 ms·
Duperemove – Tools for deduping file systems
- doctorpangloss 2y agoMicrosoft’s File Deduplication works really well. But I think they only enable it in Windows Server because it would truly cut PC profits in half if people didn’t run out of storage all the time.
- sheepdestroyer 2y agoBut Microsoft does not sell local storage?
- mrguyorama 2y agoNormal People do not run out of storage on their PCs. What data do you think normal people even have? Everything is streamed. My girlfriend has kept everything that has ever mattered to her for 15 years now, including two college degrees worth of class material, in the free tier of OneDrive, like a few gigs maybe. Her largest digital asset is a gig of mp3s that are a copy of the Harry Potter audiobooks. Even that is "streamed" from my google drive. Even getting her into the hobby of "collecting steam games" hasn't put pressure on her 500gb hard drive, because she doesn't play modern AAA games. People run out of storage on their phone from taking Video and Pictures but they don't ever copy those over to their PCs, if they even have PCs, they just buy more cloud storage space from Apple.
- floam 2y agoAnd common ways of filling up your storage, like big photo or music libraries, are unlikely to have many dupes.
- doctorpangloss 2y ago> Normal People do not run out of storage on their PCs... My girlfriend... Normal people run out of storage on their 128GB SSD macOS devices all the time.
- zeotroph 2y agoThat only seems to work on btrfs, XFS, (and maybe now or very soon) ZFS and bcachefs: "[duperemove] simply finds candidates for dedupe and submits them to the Linux kernel FIDEDUPERANGE ioctl." [1] (aka BTRFS_IOC_FILE_EXTENT_SAME), and this ioctl "performs the 'compare and share if identical'" (and locking etc.) work [2]. But on those filesystems, that is a nice feature, plus it lets the tool get away with a weak hash like murmur3. 1: http://markfasheh.github.io/duperemove/duperemove.html http://markfasheh.github.io/duperemove/duperemove.html 2: https://manpages.debian.org/bookworm/manpages-dev/ioctl_fideduperange.2.en.html https://manpages.debian.org/bookworm/manpages-dev/ioctl_fide...
- Arnavion 2y agoA fallback to hardlinking would work for a lot more filesystems, though of course it would be limited to files that are completely identical rather than any identical extents / blocks within files.
- gavinsyancey 2y agoMore importantly, hardlinking means writes to one file would affect both, while FIDUPERANGE creates copy-on-write semantics.
- JackSlateur 2y agoQuick note : I removed murmur3 support, and replaced it with xxhash
- Alifatisk 2y agoI have recently stumbled upon a challenge with deduping lots of media. I'm talking about both images and pictures. They are named in different ways, comes in different image and video formats, and are spread out throughout a whole directory and deeply nested. In total, it's about 170 GB of content. My approach have been this: - Flatten out all the deeply nested folders - Categorize the content into folders with the year as name - Run deduplication and set the threshold to 100% match (I really don't want to loose any content) - Sync to Google photos Have anyone else faced similar challenges?
- mchan 2y agoYou might be interested in this app: https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka
- 0cf8612b2e1e 2y agoIf you are only willing to drop exact duplicates, why not just take a file hash of everything as your starting point? Delete all but one of the collisions.
- Alifatisk 2y agoDupeGuru has been pretty effective solution for this. I don’t know if creating a hash for every file and then checking for collision is any better way? Sounds like it requires more labor.
- 0cf8612b2e1e 2y agofind . -type f -exec sha256sum {} \; | sort Gets you half way there.
- Alifatisk 2y agoClever, and I appreciate your suggestion! But doesn’t look like it’s worth replacing DupeGuru for that approach.
- 2y ago
- fluential 2y agoI’ve been processing a lot of niche audio files often same tracks but trimmed in length - jdupe [1] worked really well. https://codeberg.org/jbruchon/jdupes https://codeberg.org/jbruchon/jdupes
- euroderf 2y agoDo you have a solution for tracks that are identical in content but differ in metadata ? Some audio tools insist on making changes.
- toomuchtodo 2y agohttps://picard.musicbrainz.org/ https://picard.musicbrainz.org/ potentially?
- abspoel 2y agoA while back I wrote a simple tool to remove or symlink duplicate files (exact matches). It's faster than fully hashing each file, maybe it's useful to some folks: https://github.com/abspoel/dedup https://github.com/abspoel/dedup