3 ms·

PAR files are your friend here

dspillett 3y ago
Now there is a blast from the past: I par2 archives being used in usenet binary groups in the early 2000s. Unfortunately they need to be considered before you need to recover the data, which my friend did not. Also/instead having multiple copies would have been useful as one may have survived better (stored in different locations in case the environment in one makes the disk degrade faster). Though neither method removes the need to occasionally test the media to make sure you have enough workable bits to recover the data.
- joshspankit 3y ago
  Sorry to hear your friend did not. Parity files are imo an wildly underrated tech for long-term storage and something that I currently use for all long-term storage. I’ve even been known to use 1% if I’m relying on ZFS/RAID and just want to “trust but verify”. Off-site and multi-device backups are still vital for anything that’s important enough but as you mention multiple copies does not remove the need to test the media. For anyone who doesn’t know already: they got popular (and probably created) for binary files posted to usenet. Usenet was a high-speed and low-completion platform where files got corrupted all the time. PARity files took the algorithm for RAID and created a new file that could live “next to” the binary files. If you had a 1GB file and 300MB of PAR/PAR2 files could could recover the complete original file even if 30% was missing or corrupted. It still works today on Windows/macOS/Linux.
  - dspillett 3y ago
    > Parity files are imo an wildly underrated tech for long-term storage and something that I currently use for all long-term storage. As a not to others: if following this method for archive purposes, be careful how you are distributing these parity files and those that they relate to. If all your files are on the same physical disk/tape the one physical disk/tape being unreadable kills the whole lot so you are only protecting yourself against errors in individual blocks. To protect against more complete media failures (i.e. a CD/DVD being unreadable) you need to split the data over several disks/tapes. This matches how the system was used on usenet, where each post (of potentially many tens or hundreds making up a large binary post) was essentially a separate bit of storage media. At this point it may just be easier to have multiple copies, though note that even with multiple copies the parity information can be useful as you can detect bit and block level corruption and in many cases correct it (whereas with two copies you can detect corruption, as the copies no longer match, but have no reliable way of knowing which copy, if any, is correct).

Now there is a blast from the past: I par2 archives being used in usenet binary groups in the early 2000s. Unfortunately they need to be considered before you need to recover the data, which my friend did not. Also/instead having multiple copies would have been useful as one may have survived better (stored in different locations in case the environment in one makes the disk degrade faster). Though neither method removes the need to occasionally test the media to make sure you have enough workable bits to recover the data.

joshspankit 3y ago
Sorry to hear your friend did not. Parity files are imo an wildly underrated tech for long-term storage and something that I currently use for all long-term storage. I’ve even been known to use 1% if I’m relying on ZFS/RAID and just want to “trust but verify”. Off-site and multi-device backups are still vital for anything that’s important enough but as you mention multiple copies does not remove the need to test the media. For anyone who doesn’t know already: they got popular (and probably created) for binary files posted to usenet. Usenet was a high-speed and low-completion platform where files got corrupted all the time. PARity files took the algorithm for RAID and created a new file that could live “next to” the binary files. If you had a 1GB file and 300MB of PAR/PAR2 files could could recover the complete original file even if 30% was missing or corrupted. It still works today on Windows/macOS/Linux.
- dspillett 3y ago
  > Parity files are imo an wildly underrated tech for long-term storage and something that I currently use for all long-term storage. As a not to others: if following this method for archive purposes, be careful how you are distributing these parity files and those that they relate to. If all your files are on the same physical disk/tape the one physical disk/tape being unreadable kills the whole lot so you are only protecting yourself against errors in individual blocks. To protect against more complete media failures (i.e. a CD/DVD being unreadable) you need to split the data over several disks/tapes. This matches how the system was used on usenet, where each post (of potentially many tens or hundreds making up a large binary post) was essentially a separate bit of storage media. At this point it may just be easier to have multiple copies, though note that even with multiple copies the parity information can be useful as you can detect bit and block level corruption and in many cases correct it (whereas with two copies you can detect corruption, as the copies no longer match, but have no reliable way of knowing which copy, if any, is correct).