4 ms·

That's exactly why I've been keeping all "duplicates" in my photo collections. They do take up a lot of space, and just today I asked in photo.stackexchange fo

That's exactly why I've been keeping all "duplicates" in my photo collections. They do take up a lot of space, and just today I asked in photo.stackexchange for backup compression techniques that can exploit inter-image similarities: https://photo.stackexchange.com/questions/132609/backup-compression-that-takes-into-account-similarities-between-images https://photo.stackexchange.com/questions/132609/backup-comp...

syntaxfree 3y ago
Suggestion: stack the images vertically or horizontally. Frequency spectrum compression schemes like JPG will see the similarity in the fine details.
- bayesianbot 3y ago
  I got really good compression using this technique with JPEG XL, I'm sure there's even a good reason why it works so well but it's been a long time and I don't seem to remember why.
- bondarchuk 3y ago
  >in the fine details Could it be possible that jpg also exploits the repetition at the wavelength of the width of a single picture, so to say? E.g. 4 pictures side-by-side with the same black dot in the center, can all 4 dots be encoded with a single sine wave (simplifying a lot here..) that has peaks at each dot?
randyrand 3y ago
most duplicates are from the same vantage point. these are not. i.e. you don't need to keep them all.
- beagle3 3y ago
  Those have been used for denouncing and super resolution for 30 years now - they are not useless. And storage is cheap, just keep them all.
  - beagle3 3y ago
    That was supposed to be denoising, not denouncing, DYAC. Just noticed, too late to Edit Now.
bick_nyers 3y ago
Tiled/stacked approach as others mention is good, and probably the best approach. Could also try doing an uncompressed format (even just .png uncompressed) or something simple like RLE then 7zip them together since 7zip is the only archive format that does inter-file (as opposed to intra-file) compression as far as I am aware. Unfortunately lossless video compression won't help here as it will compress frames individually for lossless.
- adrianN 3y ago
  Inter file compression has been solved ever since tar|gz
  - danielheath 3y ago
    Not even remotely an efficient scheme for images or video.
  - tehsauce 3y ago
    That’s for lossless compression, i think there’s special opportunities for multi image lossy
  - beagle3 3y ago
    Not so. Gzip’s window is very small - 32K in the original gzip iirc, which meant even identical copies of a 33KB file would bot help each other. Iirc it was Bzip2 that bumped that up to 1MB, and there are now compressors with larger windows - but files have also grown, it’s not a solved problem for compression utilities. It is solved for backup - but, reatic, and a few others will do that across a backup set with no “window size” limit. …. And all of that is only true for lossless, which does not include images or video.
RockRobotRock 3y ago
Stupid question. Would a block based deduplicating file system solve this?

syntaxfree 3y ago

Suggestion: stack the images vertically or horizontally. Frequency spectrum compression schemes like JPG will see the similarity in the fine details.

bayesianbot 3y ago
I got really good compression using this technique with JPEG XL, I'm sure there's even a good reason why it works so well but it's been a long time and I don't seem to remember why.
bondarchuk 3y ago
>in the fine details Could it be possible that jpg also exploits the repetition at the wavelength of the width of a single picture, so to say? E.g. 4 pictures side-by-side with the same black dot in the center, can all 4 dots be encoded with a single sine wave (simplifying a lot here..) that has peaks at each dot?

randyrand 3y ago

most duplicates are from the same vantage point. these are not. i.e. you don't need to keep them all.

beagle3 3y ago
Those have been used for denouncing and super resolution for 30 years now - they are not useless. And storage is cheap, just keep them all.
- beagle3 3y ago
  That was supposed to be denoising, not denouncing, DYAC. Just noticed, too late to Edit Now.

bick_nyers 3y ago

Tiled/stacked approach as others mention is good, and probably the best approach. Could also try doing an uncompressed format (even just .png uncompressed) or something simple like RLE then 7zip them together since 7zip is the only archive format that does inter-file (as opposed to intra-file) compression as far as I am aware. Unfortunately lossless video compression won't help here as it will compress frames individually for lossless.

adrianN 3y ago
Inter file compression has been solved ever since tar|gz
- danielheath 3y ago
  Not even remotely an efficient scheme for images or video.
- tehsauce 3y ago
  That’s for lossless compression, i think there’s special opportunities for multi image lossy
- beagle3 3y ago
  Not so. Gzip’s window is very small - 32K in the original gzip iirc, which meant even identical copies of a 33KB file would bot help each other. Iirc it was Bzip2 that bumped that up to 1MB, and there are now compressors with larger windows - but files have also grown, it’s not a solved problem for compression utilities. It is solved for backup - but, reatic, and a few others will do that across a backup set with no “window size” limit. …. And all of that is only true for lossless, which does not include images or video.

RockRobotRock 3y ago

Stupid question. Would a block based deduplicating file system solve this?