4 ms·

How many times can you copy digital media before it degrades beyond error correction? I don't know the error rate per copied bit. I think this is an excellent

How many times can you copy digital media before it degrades beyond error correction? I don't know the error rate per copied bit. I think this is an excellent place for neural networks. They can preserve vast amounts of data compactly for many data types because they statistically compress high level abstract data which can then be used to fill in regions with high error rate, although if you did that at a large scale you'd probably end up with some constant error rate fluctuating around the average true value. All indications point to the fact that we seem to be working against the unstoppable Force of entropy - indefinite error free preservation of data is ultimately impossible.

bobthepanda 7y ago
Considering we can read parchments thousands of years old, and stone tablets going back even further, perhaps we should focus efforts on copying what we can to the time-tested preserved data formats.
- FillardMillmore 7y ago
  Im going to assume this is tongue-in-cheek, but if you happen to find a way to copy Mario 64 to a stone tablet, I'd love to hear about it.
  - EwanG 7y ago
    Not THAT difficult - assuming you have enough stone tablets. I mean the program code exists. The problem I think you'd have (and true of most of these digital artifacts) is all the other hardware and software you would have to similarly encode to be able to recreate the machine and OS capable of running it...
    - danbolt 7y ago
      Not to mention a large amount of the artifacts involved to play those games aren’t necessarily documented or accessible. The Nintendo 64’s CIC chip was only reverse engineered quite recently and required a serious reverse-engineering effort. [1] [1] https://recon.cx/2015/slides/recon2015-19-mike-ryan-john-mcmaster-marshallh-Reversing-the-Nintendo-64-CIC.pdf https://recon.cx/2015/slides/recon2015-19-mike-ryan-john-mcm...
xvector 7y ago
> How many times can you copy digital media before it degrades beyond error correction? I don't know the error rate per copied bit. I am not sure I understand. Isn’t the error rate zero? And wouldn’t you be using checksums to verify perfect copies?
- allovernow 7y ago
  Can't be zero. At the very least if bits can be randomly flipped during storage from wayward radiation, they can be flipped on copy. Error correction only works if the number of flipped bits are below a threshold. But I don't know the order of magnitude of the error rate, I assume it's quite small given that computers work.
  - Spooky23 7y ago
    It can be very close. You can get durability of data equal or greater than something like Amazon S3. I’d be more worried about disasters and budget cuts than cosmic radiation.
- neurocline 7y ago
  Back in the 2007-2008 timeframe when I added the streaming install/patch/play functionality to World of Warcraft, we also made it operate as an automatic repair system - if a disk block was read that had a hash mismatch, it just downloaded the correct block from the server (and wrote the fixed block to disk, of course). I looked at a month's worth of download logs, and estimated that 1 bit per GB per year flipped on hard disk data for our customers (we had already implemented re-reads before this point, because we had the hashes in place before we did streaming download, and so were pretty sure it was bad data on disk). Even at that fairly low rate, you can imagine that this reduced daily mysterious crashes by a nice amount. And keep in mind that hard disks already have error correction, so bits on magnetic media aren't as stable as you want. And, of course, this was not data at rest, there's lots of Wow play time going on, so lots of reads. I've been told that tape is more durable than spinning hard disks; for the sake of all those enterprise backups, I hope so. As a side effect, this made me increase how much redundancy I used for data I really cared about.
Spooky23 7y ago
There are different schools of thought. And it depends on what the scope of the discussion is. If you’re talking about preserving PDF/A or PNG files, you’re probably talking about media preservation. If you’re worried that your important document written in some version of PFS:Professional Write (an obscure word processor from the 80s) being readable, conversion and access become a problem. I recall working with a group of archivists where one camp was in favor of conversion to standard formats like PDF for certain documents, another wanted to preserve documents in the original format, and still another wanted to do both. It’s a hard problem and purity of thought makes it worse.
hcs 7y ago
> How many times can you copy digital media before it degrades beyond error correction? The idea is to act as a repeater, or like dynamically refreshed RAM, detecting and amplifying the signal at each copy before it degrades too far. You still have the error rate of reading and writing each time, but it's often cheaper to get the same error rate by refreshing cheap media than by using a more permanent medium. Interleaved error correction (such as on a CD) kind of works as you describe, spreading out the error from a single physical point across the logical data so that it can be corrected by intact data elsewhere. This works because it's designed to recover from burst errors, corresponding to a localized scratch across the track. https://en.wikipedia.org/wiki/Cross-interleaved_Reed%E2%80%93Solomon_coding https://en.wikipedia.org/wiki/Cross-interleaved_Reed%E2%80%9...

bobthepanda 7y ago

Considering we can read parchments thousands of years old, and stone tablets going back even further, perhaps we should focus efforts on copying what we can to the time-tested preserved data formats.

FillardMillmore 7y ago
Im going to assume this is tongue-in-cheek, but if you happen to find a way to copy Mario 64 to a stone tablet, I'd love to hear about it.
- EwanG 7y ago
  Not THAT difficult - assuming you have enough stone tablets. I mean the program code exists. The problem I think you'd have (and true of most of these digital artifacts) is all the other hardware and software you would have to similarly encode to be able to recreate the machine and OS capable of running it...
  - danbolt 7y ago
    Not to mention a large amount of the artifacts involved to play those games aren’t necessarily documented or accessible. The Nintendo 64’s CIC chip was only reverse engineered quite recently and required a serious reverse-engineering effort. [1] [1] https://recon.cx/2015/slides/recon2015-19-mike-ryan-john-mcmaster-marshallh-Reversing-the-Nintendo-64-CIC.pdf https://recon.cx/2015/slides/recon2015-19-mike-ryan-john-mcm...

xvector 7y ago

> How many times can you copy digital media before it degrades beyond error correction? I don't know the error rate per copied bit. I am not sure I understand. Isn’t the error rate zero? And wouldn’t you be using checksums to verify perfect copies?

allovernow 7y ago
Can't be zero. At the very least if bits can be randomly flipped during storage from wayward radiation, they can be flipped on copy. Error correction only works if the number of flipped bits are below a threshold. But I don't know the order of magnitude of the error rate, I assume it's quite small given that computers work.
- Spooky23 7y ago
  It can be very close. You can get durability of data equal or greater than something like Amazon S3. I’d be more worried about disasters and budget cuts than cosmic radiation.
neurocline 7y ago
Back in the 2007-2008 timeframe when I added the streaming install/patch/play functionality to World of Warcraft, we also made it operate as an automatic repair system - if a disk block was read that had a hash mismatch, it just downloaded the correct block from the server (and wrote the fixed block to disk, of course). I looked at a month's worth of download logs, and estimated that 1 bit per GB per year flipped on hard disk data for our customers (we had already implemented re-reads before this point, because we had the hashes in place before we did streaming download, and so were pretty sure it was bad data on disk). Even at that fairly low rate, you can imagine that this reduced daily mysterious crashes by a nice amount. And keep in mind that hard disks already have error correction, so bits on magnetic media aren't as stable as you want. And, of course, this was not data at rest, there's lots of Wow play time going on, so lots of reads. I've been told that tape is more durable than spinning hard disks; for the sake of all those enterprise backups, I hope so. As a side effect, this made me increase how much redundancy I used for data I really cared about.

Spooky23 7y ago

There are different schools of thought. And it depends on what the scope of the discussion is. If you’re talking about preserving PDF/A or PNG files, you’re probably talking about media preservation. If you’re worried that your important document written in some version of PFS:Professional Write (an obscure word processor from the 80s) being readable, conversion and access become a problem. I recall working with a group of archivists where one camp was in favor of conversion to standard formats like PDF for certain documents, another wanted to preserve documents in the original format, and still another wanted to do both. It’s a hard problem and purity of thought makes it worse.

hcs 7y ago

> How many times can you copy digital media before it degrades beyond error correction? The idea is to act as a repeater, or like dynamically refreshed RAM, detecting and amplifying the signal at each copy before it degrades too far. You still have the error rate of reading and writing each time, but it's often cheaper to get the same error rate by refreshing cheap media than by using a more permanent medium. Interleaved error correction (such as on a CD) kind of works as you describe, spreading out the error from a single physical point across the logical data so that it can be corrected by intact data elsewhere. This works because it's designed to recover from burst errors, corresponding to a localized scratch across the track. https://en.wikipedia.org/wiki/Cross-interleaved_Reed%E2%80%93Solomon_coding https://en.wikipedia.org/wiki/Cross-interleaved_Reed%E2%80%9...