17 ms·

> SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium That's what the developers of subversion thought, but

by hannob 4y ago

> SHA-1 is still perfectly fine for some applications like detecting duplicate files on a storage medium That's what the developers of subversion thought, but they didn't anticipate that once colliding files were available people would commit them to SVN repos as test cases. And then everything broke: https://www.bleepingcomputer.com/news/security/sha1-collision-attack-makes-its-first-victim-subversion-repositories/ https://www.bleepingcomputer.com/news/security/sha1-collisio...

TheRealPomax 4y ago
That changes the parameters quite a bit though. For local digests, like image deduplication of your own content, on your own computers, sha-1 is still perfectly fine. Heck, even MD5 is still workable (although more prone to collide). Nowhere in that process is the internet, or "users" or anything else like that involved =) You use digests to quickly detect potential collisions, then you verify each collision report, then you delete the actual duplicates. Human involvement still very much required because you're curating your own data.
- dylan604 4y ago
  If we're talking specifically image deduplication, then a hash comparison is only going to find you exact matches. what about the image deduplication of trying to find alt versions of things like scaling, different codecs, etc? if you want to dedupe images, some sort of phashing would be much better so that the actual image is considered vs just the specific bits to generate the image.
  - dreamcompiler 4y ago
    Very true but if the hash matches the images are guaranteed to match too. That's my first pass when deduping my drives. My second pass is looking for "logically equivalent" images.
  - TheRealPomax 4y ago
    Depends on the images. For photographs, a digest is enough. For "random images downloaded from the web", or when you're deduplicating lots of user's data, sure, you want data appropriate digests, like SIFT prints. But then we're back to "you had to bring network content back into this" =)

by hannob 4y ago

TheRealPomax 4y ago

That changes the parameters quite a bit though. For local digests, like image deduplication of your own content, on your own computers, sha-1 is still perfectly fine. Heck, even MD5 is still workable (although more prone to collide). Nowhere in that process is the internet, or "users" or anything else like that involved =) You use digests to quickly detect potential collisions, then you verify each collision report, then you delete the actual duplicates. Human involvement still very much required because you're curating your own data.

dylan604 4y ago
If we're talking specifically image deduplication, then a hash comparison is only going to find you exact matches. what about the image deduplication of trying to find alt versions of things like scaling, different codecs, etc? if you want to dedupe images, some sort of phashing would be much better so that the actual image is considered vs just the specific bits to generate the image.
- dreamcompiler 4y ago
  Very true but if the hash matches the images are guaranteed to match too. That's my first pass when deduping my drives. My second pass is looking for "logically equivalent" images.
- TheRealPomax 4y ago
  Depends on the images. For photographs, a digest is enough. For "random images downloaded from the web", or when you're deduplicating lots of user's data, sure, you want data appropriate digests, like SIFT prints. But then we're back to "you had to bring network content back into this" =)