4 ms·

Deduplicating by SHA (so, exactly the same bytes) is straightforward, and phototprism does that: https://docs.photoprism.app/user-guide/library/duplicates/ http

Deduplicating by SHA (so, exactly the same bytes) is straightforward, and phototprism does that: https://docs.photoprism.app/user-guide/library/duplicates/ https://docs.photoprism.app/user-guide/library/duplicates/ Unfortunately, if you've used Google Takeout or other systems that can both downsample your photos and videos, as well as actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics. My first approach was to build a UID based on a series of metadata fields (like captured-at time, the shutter count of the camera body, and the geohash), but that breaks when metadata is missing or edited from their sibling variants. Just finding what assets to compare against by time proved to be problematic in some edge cases. By storing in UTC (which is how I've built systems for the last thirty years!) made any file that didn't encode a time zones be wrong by whatever the correct offset was--almost all videos, and many RAW file formats don't have timezones. The solution I came up with was to encoding captured-at time _in the local time_, rather than UTC. I also infer or extract a milliseconds of precision for the date stamp, so files with more or less precision will still overlap during the initial captured-at query. If you're interested, more details are here: https://photostructure.com/faq/what-do-you-mean-by-deduplicate/ https://photostructure.com/faq/what-do-you-mean-by-deduplica... My main point here is to expose how deep this rabbit hole goes—I'd suggest you think about these different scenarios, and how aggressive or conservative you want to be in duplicate aggregation, and what you want to do with inexact duplicates. Most importantly: have full offline backups before you use any deduping or asset management system. When (not if) the tool you're using behaves differently from how you expected, you won't have lost any data.

rivo 3y ago
And if you just want to go by the pixel data, look into "perceptual hashing". https://github.com/rivo/duplo https://github.com/rivo/duplo works quite well for me, even when dealing with watermarks or slight colour correction / sharpening. You could even go further and improve your success rate with Neural Hash or something similar.
- mceachen 3y ago
  Unfortunately, (almost all!) image hashing don't detect color differences--they map images to greyscale first. This may be fine for many situations, but it will return the same result for a sepia tint, a full color original with incorrect white balance, and the final result you made after mucking with channels for a couple minutes. I also found that there really isn't one "best" image hash algorithm. Using _several different_ image hash algos turns out to be only fractionally more expensive during both compute and query times, and substantially improves both precision and recall. I'm using a mean hash, gradient diff, and a DCT, all rendered from all three CIELAB-based layers, so they're sensitive to both brightness and color differences.
  - rivo 3y ago
    The library I posted uses colour information. It won't map to greyscale first.
- eviks 3y ago
  is there an option to just calculate image hash (but on image data, not the full file of image data + metadata) without any transforms? So that if it matches you can be 100% certain it's the same image
j1elo 3y ago
Deduplicating by SHA, or by exactly identical metadata, sounds so rudimentary in 2023... 10 years ago VisiPics was already doing an excellent job to help find similar or identical images in a collection, based on visual similarity! http://www.visipics.info/ http://www.visipics.info/ So much time later, I would hope that we expect more sophisticated and flexible tools, not less, for that task.
- Bloating 3y ago
  I was trying to remember the name of this. Worked well, from what I remember
buildbot 3y ago
I went about depdup in a similar but opposite way then you did - https://github.com/mgolub2/photodb https://github.com/mgolub2/photodb I used libraw to read the actual raw data from my images, ignoring possible metadata that can get changed/updated by Capture One for example. The raw data is just fed into a hashing function, to get an exact content hash. Does not work if your image is down-sampled of course, but that was actually my goal - I want to know if the raw content has changed/bit flipped, but don't care if the metadata is altered or missing.
- mceachen 3y ago
  Ah! In this case, you might be happy to discover that ExifTool just added an "image content" hash--it's the SHA (or MD5 or whatever) hash of the _actual image bytes_, after ignoring the metadata header (and possible footer) payloads. So if you, say, change the rotation of your RAW image file (which normally will just add or change the EXIF `Orientation` tag), the image content hash will stay the same. See https://exiftool.org/ExifTool.html#ImageHashType https://exiftool.org/ExifTool.html#ImageHashType (of course, your current approach should prove substantively faster, but will only work with the formats that libraw supports)
  - dcow 3y ago
    Why is GP's approach substantially faster?
    - mceachen 3y ago
      His tool is in rust, so he can directly call the libraw function. ExifTool is a perl tool, so it'd be an external process that he'd need to coordinate with.
michaelteter 3y ago
If there was ever an obvious use for AI, identifying (or at least scoring likeliness) images should be one. Heck, we could even brute force this by resizing images via different methods and comparing checksums. But _surely_ AI can help us here.
- mceachen 3y ago
  Certainly, but it depends on your goal. Models like CLIP (https://openai.com/research/clip https://openai.com/research/clip) are designed to help describe images with an embedding, but are trained to give similar embeddings for similar image _content_, not for similar _pixels_. Simple image hashes based on small thumbnails can do similar image detection quite well (see sibling comments for links to several different approaches).
taneq 3y ago
> Most importantly: have full offline backups Do this anyway. Always.
groby_b 3y ago
If software is deduplicating images by SHA including metadata, it's missing a lot about photography workflows :) So, huge props (and I mean that - so many tools get that wrong!) on your approach of trying to identify asset groups and trying to ID members via a number of heuristics. If you want to go deeper, I suggest grouping everything by image content - that means, at the very least, comparing images via a resolution independent hash. (E.g. average down to e.g. an 8x8 picture in greyscale for the simplest approach). (undouble does that nicely, and has a number of different approaches: https://erdogant.github.io/undouble/pages/html/index.html https://erdogant.github.io/undouble/pages/html/index.html)
eviks 3y ago
> actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics Why not calculate hashes just for the image data and then do metadata conflict resolution separately?
BenTheElder 3y ago
> Unfortunately, if you've used Google Takeout or other systems that can both downsample your photos and videos, as well as actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics. I thought takeout exported the original files? (alongside metadata entered into the application)

rivo 3y ago

And if you just want to go by the pixel data, look into "perceptual hashing". https://github.com/rivo/duplo https://github.com/rivo/duplo works quite well for me, even when dealing with watermarks or slight colour correction / sharpening. You could even go further and improve your success rate with Neural Hash or something similar.

mceachen 3y ago
Unfortunately, (almost all!) image hashing don't detect color differences--they map images to greyscale first. This may be fine for many situations, but it will return the same result for a sepia tint, a full color original with incorrect white balance, and the final result you made after mucking with channels for a couple minutes. I also found that there really isn't one "best" image hash algorithm. Using _several different_ image hash algos turns out to be only fractionally more expensive during both compute and query times, and substantially improves both precision and recall. I'm using a mean hash, gradient diff, and a DCT, all rendered from all three CIELAB-based layers, so they're sensitive to both brightness and color differences.
- rivo 3y ago
  The library I posted uses colour information. It won't map to greyscale first.
eviks 3y ago
is there an option to just calculate image hash (but on image data, not the full file of image data + metadata) without any transforms? So that if it matches you can be 100% certain it's the same image

j1elo 3y ago

Deduplicating by SHA, or by exactly identical metadata, sounds so rudimentary in 2023... 10 years ago VisiPics was already doing an excellent job to help find similar or identical images in a collection, based on visual similarity! http://www.visipics.info/ http://www.visipics.info/ So much time later, I would hope that we expect more sophisticated and flexible tools, not less, for that task.

Bloating 3y ago
I was trying to remember the name of this. Worked well, from what I remember

buildbot 3y ago

I went about depdup in a similar but opposite way then you did - https://github.com/mgolub2/photodb https://github.com/mgolub2/photodb I used libraw to read the actual raw data from my images, ignoring possible metadata that can get changed/updated by Capture One for example. The raw data is just fed into a hashing function, to get an exact content hash. Does not work if your image is down-sampled of course, but that was actually my goal - I want to know if the raw content has changed/bit flipped, but don't care if the metadata is altered or missing.

mceachen 3y ago
Ah! In this case, you might be happy to discover that ExifTool just added an "image content" hash--it's the SHA (or MD5 or whatever) hash of the _actual image bytes_, after ignoring the metadata header (and possible footer) payloads. So if you, say, change the rotation of your RAW image file (which normally will just add or change the EXIF `Orientation` tag), the image content hash will stay the same. See https://exiftool.org/ExifTool.html#ImageHashType https://exiftool.org/ExifTool.html#ImageHashType (of course, your current approach should prove substantively faster, but will only work with the formats that libraw supports)
- dcow 3y ago
  Why is GP's approach substantially faster?
  - mceachen 3y ago
    His tool is in rust, so he can directly call the libraw function. ExifTool is a perl tool, so it'd be an external process that he'd need to coordinate with.

michaelteter 3y ago

If there was ever an obvious use for AI, identifying (or at least scoring likeliness) images should be one. Heck, we could even brute force this by resizing images via different methods and comparing checksums. But _surely_ AI can help us here.

mceachen 3y ago
Certainly, but it depends on your goal. Models like CLIP (https://openai.com/research/clip https://openai.com/research/clip) are designed to help describe images with an embedding, but are trained to give similar embeddings for similar image _content_, not for similar _pixels_. Simple image hashes based on small thumbnails can do similar image detection quite well (see sibling comments for links to several different approaches).

taneq 3y ago

> Most importantly: have full offline backups Do this anyway. Always.

groby_b 3y ago

If software is deduplicating images by SHA including metadata, it's missing a lot about photography workflows :) So, huge props (and I mean that - so many tools get that wrong!) on your approach of trying to identify asset groups and trying to ID members via a number of heuristics. If you want to go deeper, I suggest grouping everything by image content - that means, at the very least, comparing images via a resolution independent hash. (E.g. average down to e.g. an 8x8 picture in greyscale for the simplest approach). (undouble does that nicely, and has a number of different approaches: https://erdogant.github.io/undouble/pages/html/index.html https://erdogant.github.io/undouble/pages/html/index.html)

eviks 3y ago

> actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics Why not calculate hashes just for the image data and then do metadata conflict resolution separately?

BenTheElder 3y ago

> Unfortunately, if you've used Google Takeout or other systems that can both downsample your photos and videos, as well as actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics. I thought takeout exported the original files? (alongside metadata entered into the application)