4 ms·

Can it remove duplicates? That’s the holy grail along with storing my images. I’ve got so many damn photos, and I want to reduce the total amount I have but goi

Can it remove duplicates? That’s the holy grail along with storing my images. I’ve got so many damn photos, and I want to reduce the total amount I have but going through them is so daunting I’ll never do it without a computer-assisted organizer.

jonathankoren 3y ago
I wrote some scripts a few years ago to do just that. I haven’t used them myself in a while, but they should still be good. https://github.com/jonathankoren/photo-autorganize https://github.com/jonathankoren/photo-autorganize
mceachen 3y ago
Deduplicating by SHA (so, exactly the same bytes) is straightforward, and phototprism does that: https://docs.photoprism.app/user-guide/library/duplicates/ https://docs.photoprism.app/user-guide/library/duplicates/ Unfortunately, if you've used Google Takeout or other systems that can both downsample your photos and videos, as well as actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics. My first approach was to build a UID based on a series of metadata fields (like captured-at time, the shutter count of the camera body, and the geohash), but that breaks when metadata is missing or edited from their sibling variants. Just finding what assets to compare against by time proved to be problematic in some edge cases. By storing in UTC (which is how I've built systems for the last thirty years!) made any file that didn't encode a time zones be wrong by whatever the correct offset was--almost all videos, and many RAW file formats don't have timezones. The solution I came up with was to encoding captured-at time _in the local time_, rather than UTC. I also infer or extract a milliseconds of precision for the date stamp, so files with more or less precision will still overlap during the initial captured-at query. If you're interested, more details are here: https://photostructure.com/faq/what-do-you-mean-by-deduplicate/ https://photostructure.com/faq/what-do-you-mean-by-deduplica... My main point here is to expose how deep this rabbit hole goes—I'd suggest you think about these different scenarios, and how aggressive or conservative you want to be in duplicate aggregation, and what you want to do with inexact duplicates. Most importantly: have full offline backups before you use any deduping or asset management system. When (not if) the tool you're using behaves differently from how you expected, you won't have lost any data.
- rivo 3y ago
  And if you just want to go by the pixel data, look into "perceptual hashing". https://github.com/rivo/duplo https://github.com/rivo/duplo works quite well for me, even when dealing with watermarks or slight colour correction / sharpening. You could even go further and improve your success rate with Neural Hash or something similar.
  - mceachen 3y ago
    Unfortunately, (almost all!) image hashing don't detect color differences--they map images to greyscale first. This may be fine for many situations, but it will return the same result for a sepia tint, a full color original with incorrect white balance, and the final result you made after mucking with channels for a couple minutes. I also found that there really isn't one "best" image hash algorithm. Using _several different_ image hash algos turns out to be only fractionally more expensive during both compute and query times, and substantially improves both precision and recall. I'm using a mean hash, gradient diff, and a DCT, all rendered from all three CIELAB-based layers, so they're sensitive to both brightness and color differences.
    - rivo 3y ago
      The library I posted uses colour information. It won't map to greyscale first.
  - eviks 3y ago
    is there an option to just calculate image hash (but on image data, not the full file of image data + metadata) without any transforms? So that if it matches you can be 100% certain it's the same image
- j1elo 3y ago
  Deduplicating by SHA, or by exactly identical metadata, sounds so rudimentary in 2023... 10 years ago VisiPics was already doing an excellent job to help find similar or identical images in a collection, based on visual similarity! http://www.visipics.info/ http://www.visipics.info/ So much time later, I would hope that we expect more sophisticated and flexible tools, not less, for that task.
agent531c 3y ago
[dead]
poulpy123 3y ago
I used successfully https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka to remove duplicate and also https://www.scootersoftware.com/ https://www.scootersoftware.com/ to merge pictures folders
jyrkesh 3y ago
I used to use DupeGuru which has some photo-specific dupe detection where you can fuzzy match image dupes based on content: https://dupeguru.voltaicideas.net/ https://dupeguru.voltaicideas.net/ But I switched over to czkawka, which has a better interface for comparing files, and seems to be a bit faster: https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka Unfortunately, neither of these are integrated into Photoprism, so you still have to do some file management outside the database before importing. I also haven't used Photoprism extensively yet (I think it's running on one of my boxes, but I haven't gotten around to setting it up), but I did find that it wasn't really built for file-based libraries. It's a little more heavyweight, but my research shows that Nextcloud Memories might be a better choice for me (it's not the first-party Nextcloud photos app, but another one put together by the community): https://apps.nextcloud.com/apps/memories https://apps.nextcloud.com/apps/memories
- robobub 3y ago
  Memories is definitely loads better, especially with Recognize for face/object/video tagging
gardenhedge 3y ago
Same. Killer feature

jonathankoren 3y ago

I wrote some scripts a few years ago to do just that. I haven’t used them myself in a while, but they should still be good. https://github.com/jonathankoren/photo-autorganize https://github.com/jonathankoren/photo-autorganize

mceachen 3y ago

Deduplicating by SHA (so, exactly the same bytes) is straightforward, and phototprism does that: https://docs.photoprism.app/user-guide/library/duplicates/ https://docs.photoprism.app/user-guide/library/duplicates/ Unfortunately, if you've used Google Takeout or other systems that can both downsample your photos and videos, as well as actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics. My first approach was to build a UID based on a series of metadata fields (like captured-at time, the shutter count of the camera body, and the geohash), but that breaks when metadata is missing or edited from their sibling variants. Just finding what assets to compare against by time proved to be problematic in some edge cases. By storing in UTC (which is how I've built systems for the last thirty years!) made any file that didn't encode a time zones be wrong by whatever the correct offset was--almost all videos, and many RAW file formats don't have timezones. The solution I came up with was to encoding captured-at time _in the local time_, rather than UTC. I also infer or extract a milliseconds of precision for the date stamp, so files with more or less precision will still overlap during the initial captured-at query. If you're interested, more details are here: https://photostructure.com/faq/what-do-you-mean-by-deduplicate/ https://photostructure.com/faq/what-do-you-mean-by-deduplica... My main point here is to expose how deep this rabbit hole goes—I'd suggest you think about these different scenarios, and how aggressive or conservative you want to be in duplicate aggregation, and what you want to do with inexact duplicates. Most importantly: have full offline backups before you use any deduping or asset management system. When (not if) the tool you're using behaves differently from how you expected, you won't have lost any data.

rivo 3y ago
And if you just want to go by the pixel data, look into "perceptual hashing". https://github.com/rivo/duplo https://github.com/rivo/duplo works quite well for me, even when dealing with watermarks or slight colour correction / sharpening. You could even go further and improve your success rate with Neural Hash or something similar.
- mceachen 3y ago
  Unfortunately, (almost all!) image hashing don't detect color differences--they map images to greyscale first. This may be fine for many situations, but it will return the same result for a sepia tint, a full color original with incorrect white balance, and the final result you made after mucking with channels for a couple minutes. I also found that there really isn't one "best" image hash algorithm. Using _several different_ image hash algos turns out to be only fractionally more expensive during both compute and query times, and substantially improves both precision and recall. I'm using a mean hash, gradient diff, and a DCT, all rendered from all three CIELAB-based layers, so they're sensitive to both brightness and color differences.
  - rivo 3y ago
    The library I posted uses colour information. It won't map to greyscale first.
- eviks 3y ago
  is there an option to just calculate image hash (but on image data, not the full file of image data + metadata) without any transforms? So that if it matches you can be 100% certain it's the same image
j1elo 3y ago
Deduplicating by SHA, or by exactly identical metadata, sounds so rudimentary in 2023... 10 years ago VisiPics was already doing an excellent job to help find similar or identical images in a collection, based on visual similarity! http://www.visipics.info/ http://www.visipics.info/ So much time later, I would hope that we expect more sophisticated and flexible tools, not less, for that task.

agent531c 3y ago

[dead]

poulpy123 3y ago

I used successfully https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka to remove duplicate and also https://www.scootersoftware.com/ https://www.scootersoftware.com/ to merge pictures folders

jyrkesh 3y ago

I used to use DupeGuru which has some photo-specific dupe detection where you can fuzzy match image dupes based on content: https://dupeguru.voltaicideas.net/ https://dupeguru.voltaicideas.net/ But I switched over to czkawka, which has a better interface for comparing files, and seems to be a bit faster: https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka Unfortunately, neither of these are integrated into Photoprism, so you still have to do some file management outside the database before importing. I also haven't used Photoprism extensively yet (I think it's running on one of my boxes, but I haven't gotten around to setting it up), but I did find that it wasn't really built for file-based libraries. It's a little more heavyweight, but my research shows that Nextcloud Memories might be a better choice for me (it's not the first-party Nextcloud photos app, but another one put together by the community): https://apps.nextcloud.com/apps/memories https://apps.nextcloud.com/apps/memories

robobub 3y ago
Memories is definitely loads better, especially with Recognize for face/object/video tagging

gardenhedge 3y ago

Same. Killer feature