The "petabyte for a century" problem statement he mentions near the top is fun: can you preserve 1 PB with better-than-even odds of all the bits coming back rig

The "petabyte for a century" problem statement he mentions near the top is fun: can you preserve 1 PB with better-than-even odds of all the bits coming back right in a century? He wrote something about how he saw the durability situation in 2010: https://queue.acm.org/detail.cfm?id=1866298 https://queue.acm.org/detail.cfm?id=1866298 . He seems to define the problem to allow for maintenance, e.g. check-and-repair operations are discussed. Broadening a little, I read it as "if you're trying to be more cautious than you usually want for commercial/academic online storage or even backup, what do you do? And would it work?" A lot (but not all) of the author's Queue piece talks about stats from online storage, which doesn't have some wins you can get if you're entirely about long-term durability. In online systems, heavy ECC seems out of favor compared to replication for performance reasons, but additional LDPC or RS at the app layer can absorb a substantial % of your volumes having problems, or a ton of random bad blocks. (In a more near-line context, Backblaze uses RS: https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect... ) Same for tape -- offline LTO is slow and drives are costly, but the cost/TB and the rated lifespan seem like advantages over HDDs for this specific goal, at least in a narrow engineering sense. A pile of cryptographic hashes that fits on one device can help you check for and localize errors without sending everything over a network/doing the full ECC dance. If the hash being broken and data tampered with is in your threat model (hey, weird things can happen in a century), you can also hash with a secret nonce you keep separate. Initially loading a PB with a good chance of no mistakes is a thing too, and durability measures don't totally address that. Maybe your multi-site strategy loads up the original on different hardware at different locations with independent software implementations, and you compare those hashes after you do it. With all that the hundred-years part is still deeply tricky in a couple ways. "Lasting a hundred years" is just technologically different from "very low error rate at 5 years." Widely used media like LTO tape seem to max out at a 30y rating, and exotic archive media like the "hardened film" at GitHub's code vault has the big disadvantage of no ecosystem to read it. So seems like you really want refreshes of some sort at intervals, and being sure a task will be done decades from now is hard (assuming high-tech civilization is around and all that--some things just have to be outside the scope of the problem for it to be meaningful). From that angle, maybe having an online copy of the data is a better investment than the pure engineering perspective would suggest: if other folks can grab a copy of the archive it has a better chance of outliving your organization. Two, a lot of unknown unknowns crop up at that timescale. A couple decades back we didn't have the experience at scale we do now, and CEEs and other causes of SDC were less on anyone's radar. We could discover something else significant after it's too late to fix. The world can also change in ways that disrupt the durability picture substantially (changing laws or disaster risks, say), short of the types of change that make the whole problem meaningless. Anyway, fun question, and if you find it fun too, you might like https://www.youtube.com/watch?v=eNliOm9NtCM https://www.youtube.com/watch?v=eNliOm9NtCM , a talk on backups from someone at Google that talks about the tape restore after the big GMail glitch and various dimensions of resilience. And I'm sure there are storage papers and Long Now-ish stuff I'm not plugged into about things like this, wouldn't mind hearing about it.