3 ms·
Can anyone recommend a method to deduplicate pdfs? The hash is often different but the content and meta data is 99.99% the same.
by ck_one 2y ago
Can anyone recommend a method to deduplicate pdfs? The hash is often different but the content and meta data is 99.99% the same.
- strangus 2y agocp?
- pixelmonkey 2y agoYou might want strip metadata before doing a comparison, using exiftool. Even though exiftool was originally written for EXIF metadata on JPGs, these days, it supports a lot of metadata standards, including PDF. This command will do it assuming you set filename=`basename your.pdf .pdf`: exiftool -all= -o ${filename}.stripped.pdf ${filename}.pdf That won't help you with small differences in the contents, but might help with small differences in metadata. Running `md5sum` on the stripped PDF should give more reliable dedupe results. I was recently working on a similar problem for JPG, RAW, and MP4 files (photo/video backup) so it is fresh in my mind.
- bob1029 2y agoI would consider rasterizing the PDFs and then hashing the resulting bitmaps.