4 ms·
pHash – An open source perceptual hash library (2013)
- thrwyoilarticle 5y agoNeural nets are just locality-sensitive hashing functions, after all.
- motohagiography 5y agoIt's funny, but as a naive reader, I was reading through with the question of why you would use DCT whether it's significantly different from python's cosine_similarity, and why you would use that instead of say, pythagorean distance, edit distance, conditional entropy, and related methods. Is it a speed/accuracy trade off? Obviously I don't have depth on these as the answer would be trivial to someone who did, but I'm wondering if there a simple breakdown of what these and other information similarity methods optimize for.
- amitport 5y ago? Any references?
- wizzwizz4 5y agoWhen they convert a large number of inputs into a small number of outputs, they're acting like a hash function. (By this metric, lossy compression algorithms are all hash functions.)
- rajnathani 5y agoSparse neural networks can be optimized by implementing them using Locality Sensitive Hashing (2019 paper): https://arxiv.org/pdf/1903.03129.pdf https://arxiv.org/pdf/1903.03129.pdf Fun fact, last week one of the main authors of the above 2019 paper spun off a company for the above approach: https://techcrunch.com/2021/08/13/thirdai-raises-6m-to-democratize-ai-to-any-hardware/ https://techcrunch.com/2021/08/13/thirdai-raises-6m-to-democ...
- blacktulip 5y agoDoes it pronounce 'P-Hash' or 'Fash'?
- inglor 5y agoIt's interesting to note that conceptually stuff like pHash (which is very useful) is similar to dimensionality reduction as we see it in machine-learning/data-science scenarios.
- igorkraw 5y agoIt's not just similar, learning a classifier is 100% identical to finding a perceptual hash onto your classes that aligns with human sensibilities
- edge17 5y agoKind of. The only point of phash is to make comparing images fast and cheap... meaning from a practical standpoint there isn't a ton of useful information in the hash. All you really care about is the final hamming distance between two hashes. I think classifiers tend to encode actual interesting information.
- robbedpeter 5y agoThey can, but neural architectures range in granularity from basic autoencoders with softmax which can be equivalent to a hash function, to convolutional networks which explicitly encode fine detail. Neural networks are a great tool that when used well can operate with explicit functions like phash- they don't have to be a black box mystery. Pick the right architecture and tolerance and you might eliminate a lot of development, maintenance, and troubleshooting.
- vortico 5y agoI've been looking for something like this to write a custom solution for finding near duplicates in image libraries. Will give it a spin!
- charles_f 5y agoYou'll probably be interested in these links which describe algorithms in simple words: http://hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html http://hackerfactor.com/blog/index.php?/archives/432-Looks-L... and http://hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html http://hackerfactor.com/blog/index.php?/archives/529-Kind-of...
- deleted 5y ago[deleted]
- mceachen 5y agoNote that pHash is GPL (if you're thinking of including it in a library or product). I evaluated a ton of different images hashing algorithms for PhotoStructure, and saw similar accuracy from pHash, dhash, and mhash. I found that an mhash triplet rendered against a L*a*b* space (rather than just looking at a brightness channel) was best. I also found too many false negatives _and false positives_ when aggregating duplicates until I also included file metadata heuristics. Pesky camera manufacturers are surprisingly inconsistent in how they encode the same metadata across, say, RAW and JPG images. I'm up to a gig of exemplar pairs that CI tests use to verify a litany of weirdnesses, like inconsistently encoding exposure information (in looking at you, Google Pixels), or GPS (iDevices), or even make, model, and serial number (several dSLR and other flagship phone manufacturers). I wrote this up with some more details if you're interested: https://photostructure.com/faq/what-do-you-mean-by-deduplicate/ https://photostructure.com/faq/what-do-you-mean-by-deduplica...
- vortico 5y agoI don't need to release the software, just using for my personal internet archive server with a few TB of images. Specifically interested in finding images that are a subset of others, or recompressed with JPG, or slightly discolored or rotated, or with text overlaid. Interesting software!
- duffyjp 5y agoI've been using this in a Rails app to detect duplicate image uploads for years. It works really well. https://github.com/westonplatter/phashion https://github.com/westonplatter/phashion
- dchuk 5y agoat a high level, how did you implement this? Are you storing the phash of each image in the database and then when new images are uploaded, do you query the database to see if any matches return? If so, how does that query actually work? Or are you using this to ensure there’s no duplicates in a batch of images being uploaded?
- duffyjp 5y agoYep, I store each hash in the DB. The ruby gem has a mechanism to test similarity to your existing hashes with a threshold so small edits still match. It's surprisingly fast too, checking against every existing hash doesn't noticeably slow down the upload. I think it's a bitwise compare, but admittedly I set that up a long time ago so I'm foggy on the details. Originally I added it because I wanted a way to automatically find the full quality image from my archive and replace the aggressively scaled down versions I used to use. Back in 2003 100KB was considered pretty big for a photo, but now it looks like garbage. I never found time to finish that part of it.
- edge17 5y agoYea, i've used phashs before to find duplicates in image datasets for training neural nets. Mainly as a high level method for cleaning up large image sets.
- ek_throwaway 5y agoThis is one of the authors: https://everipedia.org/wiki/lang_en/evan-klinger https://everipedia.org/wiki/lang_en/evan-klinger
- OrvalWintermute 5y agoI don't think it is relevant to link to doxxing information, however reprehensible the racist comments made by both parties in the fracas are.
- nacs 5y agoIt's not really doxxing -- the guy's name is right on the linked phash homepage.
- OrvalWintermute 5y agoI'm not talking about that. I am talking about the link with photographs/video of his license plate number which is considered PII.
- tambourine_man 5y agoOne of the most fun times I ever had coding was implementing a simple but quite fast and reasonably efficient image hash. Loosely based on this algorithm: https://benhoyt.com/writings/duplicate-image-detection/ https://benhoyt.com/writings/duplicate-image-detection/ Highly recommended fun exercise.
- DerekBickerton 5y agoGlad it's on Github here: https://github.com/aetilius/pHash https://github.com/aetilius/pHash Github repos tend to outlive the homepage of some projects. I got worried when I read this: > Copyright © 2008-2010
- etaioinshrdlu 5y agoI use this a lot for image hashing. How does it compare to Apple's NeuralHash? I would expect that Apple probably did a very good job at their implementation, and pHash is quite old tech.