5 ms·
> This could be super useful for deduplicating large photo datasets! That is indeed the use case mentioned in the article. A hash like this is useful for dedup
by bityard 5y ago
> This could be super useful for deduplicating large photo datasets!
That is indeed the use case mentioned in the article. A hash like this is useful for deduplicating large _file_ datasets but for deduplicating images in particular, you really want a perceptual hash. Because two images files can _look_ identical but have slightly (or wildly) different byte streams. The trade-off is that perceptual hashes are notorious for false positives in large datasets.
- buildbot 5y agoYep, just happy someone was looking into it, I currently use xxHash for this. Perceptual hashing for photos isn't great in my opinion, for example I have many raw photos that look very similar, the only difference may be a slightly higher shutter speed, or a correction of the camera angle. I'm guessing many perceptual hashes would mark these as duplicate. Maybe that's what many want, it's not what I want personally.
- pepoluan 5y agoAnd what's wrong with xxHash? According to xxHash's performance comparison page, xxHash is faster than MeowHash.
- kayamon 5y agoI once saw a guy dedupe a folder by batch-converting them to low quality JPEG thumbnails then sorting by size.