3 ms·
Over the years I've used many, many tools intended to solve this problem. In the end, after much frustration, I just use existing tools, glued together in a un
by ColinWright 5y ago
Over the years I've used many, many tools intended to solve this problem. In the end, after much frustration, I just use existing tools, glued together in a un*x manner.
find * -type f -exec md5sum '{}' ';' \
| tee /tmp/index_file.txt \
| gawk '{print $1}' \
| sort | uniq -c \
| gawk '/^ *1 /{ print $2 } \
> /tmp/duplicates.txt
for m in $( cat /tmp/duplicates.txt )
do
grep $m /tmp/index_file.txt
echo ========
done \
| less
Tweak as necessary. I do have a comparison executable that only compares sizes and sub-portions to save time, but I generally find it's not worth it.
It takes less time to type this that than it does to remember what some random other tool is called, or how to use it. I also have saved a variant that identifies similar files, and another that identifies directory structures with lots of shared files, but those are (understandably) more complex (and fragile).
- pdimitar 5y agoCan you share your other scripts? They sound exactly like what I need lately!
- ColinWright 5y agoI haven't used them for quite some time, so I'll have to dig them out. I'll also have to remember their limitations so you can better assess whether they're useful as stands, or need to be used purely as inspiration. I'll check on it/them.
- pkolaczk 5y agoMost programs I tested have very simple basic usage - just the program name and a list of directories. I doubt typing the above would be faster, and figuring out for the first time - definitely not. Also executing that on a million of files would take ages, even compared to slowest proper duplicate finders. Anyway, thanks for sharing - it is always very exciting to see how far you can go with a few unix utilities and a bit of scripting :)
- mikst 5y agoYou need to check for empty files find ... \! -empty ... they have the same hash, but they do not need to be treated as duplicate