4 ms·

I've been looking for something like this to write a custom solution for finding near duplicates in image libraries. Will give it a spin!

charles_f 5y ago
You'll probably be interested in these links which describe algorithms in simple words: http://hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html http://hackerfactor.com/blog/index.php?/archives/432-Looks-L... and http://hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html http://hackerfactor.com/blog/index.php?/archives/529-Kind-of...
deleted 5y ago
[deleted]
mceachen 5y ago
Note that pHash is GPL (if you're thinking of including it in a library or product). I evaluated a ton of different images hashing algorithms for PhotoStructure, and saw similar accuracy from pHash, dhash, and mhash. I found that an mhash triplet rendered against a L*a*b* space (rather than just looking at a brightness channel) was best. I also found too many false negatives _and false positives_ when aggregating duplicates until I also included file metadata heuristics. Pesky camera manufacturers are surprisingly inconsistent in how they encode the same metadata across, say, RAW and JPG images. I'm up to a gig of exemplar pairs that CI tests use to verify a litany of weirdnesses, like inconsistently encoding exposure information (in looking at you, Google Pixels), or GPS (iDevices), or even make, model, and serial number (several dSLR and other flagship phone manufacturers). I wrote this up with some more details if you're interested: https://photostructure.com/faq/what-do-you-mean-by-deduplicate/ https://photostructure.com/faq/what-do-you-mean-by-deduplica...
- vortico 5y ago
  I don't need to release the software, just using for my personal internet archive server with a few TB of images. Specifically interested in finding images that are a subset of others, or recompressed with JPG, or slightly discolored or rotated, or with text overlaid. Interesting software!

charles_f 5y ago

You'll probably be interested in these links which describe algorithms in simple words: http://hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html http://hackerfactor.com/blog/index.php?/archives/432-Looks-L... and http://hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html http://hackerfactor.com/blog/index.php?/archives/529-Kind-of...

deleted 5y ago

[deleted]

mceachen 5y ago

Note that pHash is GPL (if you're thinking of including it in a library or product). I evaluated a ton of different images hashing algorithms for PhotoStructure, and saw similar accuracy from pHash, dhash, and mhash. I found that an mhash triplet rendered against a L*a*b* space (rather than just looking at a brightness channel) was best. I also found too many false negatives _and false positives_ when aggregating duplicates until I also included file metadata heuristics. Pesky camera manufacturers are surprisingly inconsistent in how they encode the same metadata across, say, RAW and JPG images. I'm up to a gig of exemplar pairs that CI tests use to verify a litany of weirdnesses, like inconsistently encoding exposure information (in looking at you, Google Pixels), or GPS (iDevices), or even make, model, and serial number (several dSLR and other flagship phone manufacturers). I wrote this up with some more details if you're interested: https://photostructure.com/faq/what-do-you-mean-by-deduplicate/ https://photostructure.com/faq/what-do-you-mean-by-deduplica...

vortico 5y ago
I don't need to release the software, just using for my personal internet archive server with a few TB of images. Specifically interested in finding images that are a subset of others, or recompressed with JPG, or slightly discolored or rotated, or with text overlaid. Interesting software!