3 ms·

This could be super useful for deduplicating large photo datasets! It might be worth changing this link to the github, there’s a lot more info on the hash ther

This could be super useful for deduplicating large photo datasets! It might be worth changing this link to the github, there’s a lot more info on the hash there.

bityard 5y ago
> This could be super useful for deduplicating large photo datasets! That is indeed the use case mentioned in the article. A hash like this is useful for deduplicating large _file_ datasets but for deduplicating images in particular, you really want a perceptual hash. Because two images files can _look_ identical but have slightly (or wildly) different byte streams. The trade-off is that perceptual hashes are notorious for false positives in large datasets.
- buildbot 5y ago
  Yep, just happy someone was looking into it, I currently use xxHash for this. Perceptual hashing for photos isn't great in my opinion, for example I have many raw photos that look very similar, the only difference may be a slightly higher shutter speed, or a correction of the camera angle. I'm guessing many perceptual hashes would mark these as duplicate. Maybe that's what many want, it's not what I want personally.
  - pepoluan 5y ago
    And what's wrong with xxHash? According to xxHash's performance comparison page, xxHash is faster than MeowHash.
- kayamon 5y ago
  I once saw a guy dedupe a folder by batch-converting them to low quality JPEG thumbnails then sorting by size.
NKosmatos 5y ago
Came here to say the same thing :-) Anyone know of a free(ish) software based on a very fast hashing algorith for checking large number of images? My intention is to check for bit-rot and other unforeseen data corruptions between backup disks, I assume many others must have a similar use case.
- radicality 5y ago
  I haven't used it extensively, but this is free and you can pick from a number of hash functions - https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka

This could be super useful for deduplicating large photo datasets! It might be worth changing this link to the github, there’s a lot more info on the hash there.

bityard 5y ago

> This could be super useful for deduplicating large photo datasets! That is indeed the use case mentioned in the article. A hash like this is useful for deduplicating large _file_ datasets but for deduplicating images in particular, you really want a perceptual hash. Because two images files can _look_ identical but have slightly (or wildly) different byte streams. The trade-off is that perceptual hashes are notorious for false positives in large datasets.

buildbot 5y ago
Yep, just happy someone was looking into it, I currently use xxHash for this. Perceptual hashing for photos isn't great in my opinion, for example I have many raw photos that look very similar, the only difference may be a slightly higher shutter speed, or a correction of the camera angle. I'm guessing many perceptual hashes would mark these as duplicate. Maybe that's what many want, it's not what I want personally.
- pepoluan 5y ago
  And what's wrong with xxHash? According to xxHash's performance comparison page, xxHash is faster than MeowHash.
kayamon 5y ago
I once saw a guy dedupe a folder by batch-converting them to low quality JPEG thumbnails then sorting by size.

NKosmatos 5y ago

Came here to say the same thing :-) Anyone know of a free(ish) software based on a very fast hashing algorith for checking large number of images? My intention is to check for bit-rot and other unforeseen data corruptions between backup disks, I assume many others must have a similar use case.

radicality 5y ago
I haven't used it extensively, but this is free and you can pick from a number of hash functions - https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka