Try libpuzzle : http://www.pureftpd.org/project/libpuzzle http://www.pureftpd.org/project/libpuzzle I'm using it in an erlang wrapper (https://github.com/erase

Try libpuzzle : http://www.pureftpd.org/project/libpuzzle http://www.pureftpd.org/project/libpuzzle I'm using it in an erlang wrapper (https://github.com/eraserewind/erlpuzzle https://github.com/eraserewind/erlpuzzle) to crawl tumblr & reddit images, and then dedupe it. Puzzle signatures are easy to index. I'm using ElasticSearch