3 ms·
I have recently stumbled upon a challenge with deduping lots of media. I'm talking about both images and pictures. They are named in different ways, comes in di
by Alifatisk 2y ago
I have recently stumbled upon a challenge with deduping lots of media. I'm talking about both images and pictures. They are named in different ways, comes in different image and video formats, and are spread out throughout a whole directory and deeply nested. In total, it's about 170 GB of content.
My approach have been this:
- Flatten out all the deeply nested folders
- Categorize the content into folders with the year as name
- Run deduplication and set the threshold to 100% match (I really don't want to loose any content)
- Sync to Google photos
Have anyone else faced similar challenges?
- mchan 2y agoYou might be interested in this app: https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka
- 0cf8612b2e1e 2y agoIf you are only willing to drop exact duplicates, why not just take a file hash of everything as your starting point? Delete all but one of the collisions.
- Alifatisk 2y agoDupeGuru has been pretty effective solution for this. I don’t know if creating a hash for every file and then checking for collision is any better way? Sounds like it requires more labor.
- 0cf8612b2e1e 2y agofind . -type f -exec sha256sum {} \; | sort Gets you half way there.
- Alifatisk 2y agoClever, and I appreciate your suggestion! But doesn’t look like it’s worth replacing DupeGuru for that approach.
- corndoge 2y agoI think lots of people have faced this challenge. There are several tools that will find and allow you to purge exact duplicates, such as dupeguru.
- natmaka 2y agoI may have misunderstood but why aren't standard tools (such as rdfind or fdupes) inadequate?
- Alifatisk 2y agoIt's not about them being insufficient, I just didn't know about them. Thank you!