3 ms·

Recovering failed Photos is needed. Of 170 000 photos, I've seen dozens to hundreds bit rot on each photo sync service (Amazon, Google, Apple). If Apple start

Recovering failed Photos is needed. Of 170 000 photos, I've seen dozens to hundreds bit rot on each photo sync service (Amazon, Google, Apple). If Apple started using something like today's vision capable LLMs to understand if JPEGs and RAWs are technically correct but visually corrupt, then going back to some WORM storage to restore problem images, that would be incredible. Recovering to a factory wiped hardware device under an unrelated AppleID makes no sense, and it would seem all online conversation about this may trace back to a single, possibly confused, reddit comment.

ksec 2y ago
> I've seen dozens to hundreds bit rot on each photo sync service (Amazon, Google, Apple). Hold on a sec. How did this happen? I am assuming 1. Bit rot isn't a thing on NAND. Or a probability much much lower than HDD. 2. I am also assuming this wouldn't happen on their end because they will have all the protection in place.
- Terretta 2y ago
  One would assume. But something happens (used to happen) where a file gets corrupted on cloud end, on my end, or in transfer, then it seems a sync detects a difference and assumes it must be an update, so replaces a good file with a corrupted file, and then, well, that's it for that image. Tools like this can try to help, but usually fail: https://betanews.com/2015/03/24/fix-corrupt-photos-with-jpeg-repair-shop/ https://betanews.com/2015/03/24/fix-corrupt-photos-with-jpeg... I shoot on redundant CF cards, I back those up to long term RAID backed by S3/Glacier, so someday I can re-import the ones that are "lost", but determining which those are is surprisingly difficult when what you see in the UI isn't the file on disk any more, it's a generated thumbnail, and when the corruption doesn't affect the JPEG or RAW file format integrity, it's just a few bits flipped here and there. I haven't found any CLI or GUI tools that correctly identify all of them. I haven't yet tried using vision LLMs, discouraged by the cloud LLM cost or local LLM compute time that would take. NOTE: This hasn't happened in years. Perhaps entirely coincidentally and anecdotally, hasn't happened to me since all Apple devices are on Apple Silicon with none running any Rosetta-requiring software (I don't let Rosetta install at all), implication being the entire ecosystem is now code from within the past 5 years.
- bcrl 2y ago
  Bit rot is very much an issue on NAND. Reading causes disturbances. Flash chips have to compensate for drift in the voltage levels of cells that occurs as things age. While there are layers of protection using error correcting codes, in my experience corruption is more common with SSDs than HDDs. I've retired at least 3 SSDs from various generations over the past 14 years because of data corruption, and 1 SSD that simply refused to show up anymore. In comparison, the HDDs I've had have almost all failed without returning corrupt data, mostly due to mechanical failure. Sadly, Apple doesn't believe in using ECC memory on their devices. The photos might get corrupted in RAM on the phone, or on an Apple laptop or who knows where. Without end to end checksums, there's no way to know when and where the photos are getting corrupted. Modern filesystems with built in checksumming are definitely worth the overhead.
ryan-c 2y ago
or... they could hash the images and use an error correcting code to allow recovery of some lost files?
bugbuddy 2y ago
170,000 photos. Dear god. Please tell me how you manage such a large collection. Folders by dates, locations, people? How do you search for something? What software do you use? Apple photos is not the fastest even on the latest Mac machines.
- Terretta 2y ago
  Once upon a time it was Aperture. Because Adobe's Lightroom didn't have some things I wanted, I stubbornly stayed with Aperture including using "Retroactive": https://github.com/cormiertyshawn895/Retroactive https://github.com/cormiertyshawn895/Retroactive It stayed that until Apple eventually managed to bring Photos library up to the same spec including RAW support, non destructive edits, etc., and eventually even import Aperture libraries w/o loss. Since then, stayed with Photos, and it's BnL fine if you mostly ignore "folders" and let it do dates and people and now objects search on-device. To prevent errors, I've found I must maintain at least one machine where all originals are on a RAID array.
bananapub 2y ago
> If Apple started using something like today's vision capable LLMs to understand if JPEGs and RAWs are technically correct but visually corrupt, then going back to some WORM storage to restore problem images, that would be incredible. hahaha what? they should be checksumming things, not applying hilariously irrelevant and overcomplicates software to this problem.
- Terretta 2y ago
  Sure, if that had been done from the originating device. There are photo file formats under discussion today that do this. The existing ones do not. After the fact, it's been a hard problem since what's rendered can be visually corrupted while the format remains valid. So the problem becomes to detect whether it's visually off, at scale.

ksec 2y ago

> I've seen dozens to hundreds bit rot on each photo sync service (Amazon, Google, Apple). Hold on a sec. How did this happen? I am assuming 1. Bit rot isn't a thing on NAND. Or a probability much much lower than HDD. 2. I am also assuming this wouldn't happen on their end because they will have all the protection in place.

Terretta 2y ago
One would assume. But something happens (used to happen) where a file gets corrupted on cloud end, on my end, or in transfer, then it seems a sync detects a difference and assumes it must be an update, so replaces a good file with a corrupted file, and then, well, that's it for that image. Tools like this can try to help, but usually fail: https://betanews.com/2015/03/24/fix-corrupt-photos-with-jpeg-repair-shop/ https://betanews.com/2015/03/24/fix-corrupt-photos-with-jpeg... I shoot on redundant CF cards, I back those up to long term RAID backed by S3/Glacier, so someday I can re-import the ones that are "lost", but determining which those are is surprisingly difficult when what you see in the UI isn't the file on disk any more, it's a generated thumbnail, and when the corruption doesn't affect the JPEG or RAW file format integrity, it's just a few bits flipped here and there. I haven't found any CLI or GUI tools that correctly identify all of them. I haven't yet tried using vision LLMs, discouraged by the cloud LLM cost or local LLM compute time that would take. NOTE: This hasn't happened in years. Perhaps entirely coincidentally and anecdotally, hasn't happened to me since all Apple devices are on Apple Silicon with none running any Rosetta-requiring software (I don't let Rosetta install at all), implication being the entire ecosystem is now code from within the past 5 years.
bcrl 2y ago
Bit rot is very much an issue on NAND. Reading causes disturbances. Flash chips have to compensate for drift in the voltage levels of cells that occurs as things age. While there are layers of protection using error correcting codes, in my experience corruption is more common with SSDs than HDDs. I've retired at least 3 SSDs from various generations over the past 14 years because of data corruption, and 1 SSD that simply refused to show up anymore. In comparison, the HDDs I've had have almost all failed without returning corrupt data, mostly due to mechanical failure. Sadly, Apple doesn't believe in using ECC memory on their devices. The photos might get corrupted in RAM on the phone, or on an Apple laptop or who knows where. Without end to end checksums, there's no way to know when and where the photos are getting corrupted. Modern filesystems with built in checksumming are definitely worth the overhead.

ryan-c 2y ago

or... they could hash the images and use an error correcting code to allow recovery of some lost files?

bugbuddy 2y ago

170,000 photos. Dear god. Please tell me how you manage such a large collection. Folders by dates, locations, people? How do you search for something? What software do you use? Apple photos is not the fastest even on the latest Mac machines.

Terretta 2y ago
Once upon a time it was Aperture. Because Adobe's Lightroom didn't have some things I wanted, I stubbornly stayed with Aperture including using "Retroactive": https://github.com/cormiertyshawn895/Retroactive https://github.com/cormiertyshawn895/Retroactive It stayed that until Apple eventually managed to bring Photos library up to the same spec including RAW support, non destructive edits, etc., and eventually even import Aperture libraries w/o loss. Since then, stayed with Photos, and it's BnL fine if you mostly ignore "folders" and let it do dates and people and now objects search on-device. To prevent errors, I've found I must maintain at least one machine where all originals are on a RAID array.

bananapub 2y ago

> If Apple started using something like today's vision capable LLMs to understand if JPEGs and RAWs are technically correct but visually corrupt, then going back to some WORM storage to restore problem images, that would be incredible. hahaha what? they should be checksumming things, not applying hilariously irrelevant and overcomplicates software to this problem.

Terretta 2y ago
Sure, if that had been done from the originating device. There are photo file formats under discussion today that do this. The existing ones do not. After the fact, it's been a hard problem since what's rendered can be visually corrupted while the format remains valid. So the problem becomes to detect whether it's visually off, at scale.