19 ms·

I admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2

by na85 6y ago

I admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2 files to hopefully ward off bit rot. I feel like there's some room for improvement.

pmiller2 6y ago
Do you store your par2 files on different discs from the data they refer to? Do you have multiple copies? Do you store the discs in a cool, dark place when you're not using them? Do you have another form of backup, preferably offsite? If you do all these things, I think that's about the best you can do with optical media.
- na85 6y ago
  Haha, I keep the par2s on the same disc, and then I make multiple copies of the same disc and they're in a cardboard box in my basement somewhere.
  - jordanthoms 6y ago
    The ink on writable optical disks fades over time... So of all the copies were written at the same time, they might all stop working at around the same time and cause problems :)
    - Widdershin 6y ago
      Yeah if you're really taking this seriously it's worth investing in an LTO-4 drive and some tapes. There might be a better medium available nowadays but if I seriously wanted to have a piece of data fifty years from now that's where I'd start.
      - pmiller2 6y ago
        Really, any second form of backup would be a good idea. Preferably, it should be stored offsite and separate from any other backups.
denkmoon 6y ago
You can buy a 2tb drive for $100. $200 and you've got 430 DVD's worth of data redundantly stored. $300 and you've got local redundancy AND offsite backup.
- paranoidrobot 6y ago
  Even if magnetically they don't have 'bit rot', they use bearings where the lubrication can dry up and wear out when they're not turning for long periods of time. You need to keep them spinning on a regular basis, and replace them as they begin to fail.
  - dannyw 6y ago
    HDDs are also prone to silent bitrot, where it will simply return incorrect bytes for a sector, even without any smart errors. (Optical disks also bitrot; but so does HDDs). This is usually a precursor to SMART errors happening in the near future, but unfortunately, it can still result in corrupted replication and corrupted backups; as your backups would be backing up the rotten (corrupt) data. I've witnessed this happen on both Seagate and WD drives, on systems with ECC memory. I can only suspect this is due to HDD manufacturers wanting to reduce their error rates, and RMA rates: it may happen with the ECC bits in a sector is corrupt, making bitrot undetectable. Instead of giving an error (and being grounds for a RMA replacement), the HDD firmware may choose to return non-integrity-checked data; which would usually be correct but also could be corrupt. It's why filesystems like ZFS and btrfs are so important. My rough estimation of this, based on my own experiences and those on r/DataHoarder, suggests 1 hardware sector (4KB for most drives post 2011) will silently corrupt per 10TB year. Such corruption can be detected via checksumming disc systems like ZFS. Usually, the whole sector is garbage, which is not indicative of cosmic ray bitflips. External flash memory storage like USB sticks and SD cards fare far worse. In my own experience, silent corruption occurs more like 1 file per device, per 2-3 years; irrespective of the size of memory. I've had USB sticks and SD cards return bogus data without errors, so often. I only know because I checksum everything, otherwise I would have thought the artefacts in my videos or photos came with the source. If, in 2020, you are not using ZFS or btrfs for long term archival, you are doing something wrong. ext4, NTFS, APFS, etc may be tried and tested, but they have no checksumming, and that is a problem.
Smoosh 6y ago
You can always try an affordable online service like Backblaze B2 as one of your options. I haven't checked lately but I think they cost around $5 per terabyte per month. Plus usage, but that would be minimal for an archive.
- dannyw 6y ago
  Your credit card may expire or get blocked, and you might be in hospital, or billing alert emails may simply go to your spam folder for 45 days. Your data will be irreversibly deleted by Backblaze. There are LOTS of failure cases with any cloud provider, especially one with a crazy policy of deleting data in just 45 days. There is at least 1 reddit post a month about how someone lost data with Backblaze. Their reddit support rep is never able to do anything about it, other than "sorry, we will take on board your feedback". For comparison, if your Google Drive subscription lapses, Google stops you from uploading but will not delete your data.
  - deleted 6y ago
    [deleted]
  - lostlogin 6y ago
    This. I was backing up an external drive on a very very slow connection. I plugged it in to the machine late in the month, but forgot to turn the power on at the wall. The upload didn’t start and I lost the full backup and had to Re-upload the whole drive. A good lesson was learned but it hurt. The upload took weeks to complete.
kstenerud 6y ago
Hook up ZRaid SSDs on a low powered machine (like raspberry pi). Set up regular ZFS scrubbing, and connect that to a monitoring service. You're not going to prevent silent bitrot no matter what modern technology you use, so take a proactive approach instead to prevent data loss.
- mnw21cam 6y ago
  Agreed. I used to back up onto DVDs, creating a set of (say) 12 DVDs with an extra couple generated by par2, to take care of the case where some of the DVDs just straight aren't readable. However, I found that I had a lot of data to back up, and it was actually cheaper and less tedious to get 4TB USB hard drives for ~£100 each, and plug them into an old defunct EeePC901 (with the added advantage that if the power goes out it has a battery). My main PC has an SSH private key that lets it access a restricted shell on the EeePC that only allows it to give it files to store. That way, if a hacker breaks into my internet-facing machine, all they can do to my backups is fill the disc up, not delete or access anything. I have a process on the EeePC that regularly scrubs the par2 files, and the hard drives (I have two so far) are formatted with BTRFS, so given all the data is regularly read by the scrub process, that should notice any drive failures. My main PC uses ZFS, so I have safety in variety. I also have an off-site backup stored on an encrypted USB hard drive in my locked locker at work, which isn't updated as regularly. My internet connection is slow, so I use the rsync --only-write-batch trick, and then carry the large update file to the backup on my laptop. What could possibly go wrong?
  - lostlogin 6y ago
    Re: 4Tb drives, I do the dollars-per-mb calculation before buying hard drives. The most recent time I included the enclosure cost and found that it was actually cheaper to go huge. Granted the enclosure was a Synology, but buying 16TB drives is the closest I’ve been so ‘solving’ storage in a long time. Formatting them and adding them to the array was brutal, and they are noisy, but it has been worth it.
    - mnw21cam 6y ago
pwg 6y ago
If you continue your DVD method you may also want to look into "dvdisaster" https://en.wikipedia.org/wiki/Dvdisaster https://en.wikipedia.org/wiki/Dvdisaster
zozbot234 6y ago
Most BD-R discs are physically higher quality than DVD-R's. They can be comparable to archival-quality M-Disc media.
tarruda 6y ago
I suggest trying syncthing. It is a cross platform P2P file synchronization tool, super easy to setup.
brnt 6y ago
Anyone who uses par2 to protect a filetree, I wrote a small utility to help you maintain that filetree (e.g. bulk verify/create/status) [1]. [1] https://github.com/brenthuisman/par2deep https://github.com/brenthuisman/par2deep

by na85 6y ago

pmiller2 6y ago

Do you store your par2 files on different discs from the data they refer to? Do you have multiple copies? Do you store the discs in a cool, dark place when you're not using them? Do you have another form of backup, preferably offsite? If you do all these things, I think that's about the best you can do with optical media.

na85 6y ago
Haha, I keep the par2s on the same disc, and then I make multiple copies of the same disc and they're in a cardboard box in my basement somewhere.
- jordanthoms 6y ago
  The ink on writable optical disks fades over time... So of all the copies were written at the same time, they might all stop working at around the same time and cause problems :)
  - Widdershin 6y ago
    Yeah if you're really taking this seriously it's worth investing in an LTO-4 drive and some tapes. There might be a better medium available nowadays but if I seriously wanted to have a piece of data fifty years from now that's where I'd start.
    - pmiller2 6y ago
      Really, any second form of backup would be a good idea. Preferably, it should be stored offsite and separate from any other backups.

denkmoon 6y ago

You can buy a 2tb drive for $100. $200 and you've got 430 DVD's worth of data redundantly stored. $300 and you've got local redundancy AND offsite backup.

paranoidrobot 6y ago
Even if magnetically they don't have 'bit rot', they use bearings where the lubrication can dry up and wear out when they're not turning for long periods of time. You need to keep them spinning on a regular basis, and replace them as they begin to fail.
- dannyw 6y ago
  HDDs are also prone to silent bitrot, where it will simply return incorrect bytes for a sector, even without any smart errors. (Optical disks also bitrot; but so does HDDs). This is usually a precursor to SMART errors happening in the near future, but unfortunately, it can still result in corrupted replication and corrupted backups; as your backups would be backing up the rotten (corrupt) data. I've witnessed this happen on both Seagate and WD drives, on systems with ECC memory. I can only suspect this is due to HDD manufacturers wanting to reduce their error rates, and RMA rates: it may happen with the ECC bits in a sector is corrupt, making bitrot undetectable. Instead of giving an error (and being grounds for a RMA replacement), the HDD firmware may choose to return non-integrity-checked data; which would usually be correct but also could be corrupt. It's why filesystems like ZFS and btrfs are so important. My rough estimation of this, based on my own experiences and those on r/DataHoarder, suggests 1 hardware sector (4KB for most drives post 2011) will silently corrupt per 10TB year. Such corruption can be detected via checksumming disc systems like ZFS. Usually, the whole sector is garbage, which is not indicative of cosmic ray bitflips. External flash memory storage like USB sticks and SD cards fare far worse. In my own experience, silent corruption occurs more like 1 file per device, per 2-3 years; irrespective of the size of memory. I've had USB sticks and SD cards return bogus data without errors, so often. I only know because I checksum everything, otherwise I would have thought the artefacts in my videos or photos came with the source. If, in 2020, you are not using ZFS or btrfs for long term archival, you are doing something wrong. ext4, NTFS, APFS, etc may be tried and tested, but they have no checksumming, and that is a problem.

Smoosh 6y ago

You can always try an affordable online service like Backblaze B2 as one of your options. I haven't checked lately but I think they cost around $5 per terabyte per month. Plus usage, but that would be minimal for an archive.

dannyw 6y ago
Your credit card may expire or get blocked, and you might be in hospital, or billing alert emails may simply go to your spam folder for 45 days. Your data will be irreversibly deleted by Backblaze. There are LOTS of failure cases with any cloud provider, especially one with a crazy policy of deleting data in just 45 days. There is at least 1 reddit post a month about how someone lost data with Backblaze. Their reddit support rep is never able to do anything about it, other than "sorry, we will take on board your feedback". For comparison, if your Google Drive subscription lapses, Google stops you from uploading but will not delete your data.
- deleted 6y ago
  [deleted]
- lostlogin 6y ago
  This. I was backing up an external drive on a very very slow connection. I plugged it in to the machine late in the month, but forgot to turn the power on at the wall. The upload didn’t start and I lost the full backup and had to Re-upload the whole drive. A good lesson was learned but it hurt. The upload took weeks to complete.

kstenerud 6y ago

Hook up ZRaid SSDs on a low powered machine (like raspberry pi). Set up regular ZFS scrubbing, and connect that to a monitoring service. You're not going to prevent silent bitrot no matter what modern technology you use, so take a proactive approach instead to prevent data loss.

mnw21cam 6y ago
Agreed. I used to back up onto DVDs, creating a set of (say) 12 DVDs with an extra couple generated by par2, to take care of the case where some of the DVDs just straight aren't readable. However, I found that I had a lot of data to back up, and it was actually cheaper and less tedious to get 4TB USB hard drives for ~£100 each, and plug them into an old defunct EeePC901 (with the added advantage that if the power goes out it has a battery). My main PC has an SSH private key that lets it access a restricted shell on the EeePC that only allows it to give it files to store. That way, if a hacker breaks into my internet-facing machine, all they can do to my backups is fill the disc up, not delete or access anything. I have a process on the EeePC that regularly scrubs the par2 files, and the hard drives (I have two so far) are formatted with BTRFS, so given all the data is regularly read by the scrub process, that should notice any drive failures. My main PC uses ZFS, so I have safety in variety. I also have an off-site backup stored on an encrypted USB hard drive in my locked locker at work, which isn't updated as regularly. My internet connection is slow, so I use the rsync --only-write-batch trick, and then carry the large update file to the backup on my laptop. What could possibly go wrong?
- lostlogin 6y ago
  Re: 4Tb drives, I do the dollars-per-mb calculation before buying hard drives. The most recent time I included the enclosure cost and found that it was actually cheaper to go huge. Granted the enclosure was a Synology, but buying 16TB drives is the closest I’ve been so ‘solving’ storage in a long time. Formatting them and adding them to the array was brutal, and they are noisy, but it has been worth it.
  - mnw21cam 6y ago

pwg 6y ago

If you continue your DVD method you may also want to look into "dvdisaster" https://en.wikipedia.org/wiki/Dvdisaster https://en.wikipedia.org/wiki/Dvdisaster

zozbot234 6y ago

Most BD-R discs are physically higher quality than DVD-R's. They can be comparable to archival-quality M-Disc media.

tarruda 6y ago

I suggest trying syncthing. It is a cross platform P2P file synchronization tool, super easy to setup.

brnt 6y ago

Anyone who uses par2 to protect a filetree, I wrote a small utility to help you maintain that filetree (e.g. bulk verify/create/status) [1]. [1] https://github.com/brenthuisman/par2deep https://github.com/brenthuisman/par2deep