4 ms·

If you already have a lot of data on older, less reliable storage formats you have the choice of keeping a full copy all that data on something with higher reli

If you already have a lot of data on older, less reliable storage formats you have the choice of keeping a full copy all that data on something with higher reliability, or store just a small fraction of it in the form of the ECC file. The first option may be technically more sound but comes with a larger investment in terms of time and money. Second option may be less reliable but with significant savings. It's up to the owner of the data to decide how to balance these.

stavros 2y ago
What sort of higher-reliability options (than a Blu-ray) does a consumer have? I struggle to think of something that's not fairly niche, eg a magnetic tape drive.
- buran77 2y ago
  > higher-reliability options You'd be comparing to an old optical disc. So almost anything is higher reliability than that, starting with a newer optical disc and probably ending at any cloud storage. > does a consumer have This isn't limited just to your average consumer. DVDs are still in use in libraries and different archives which can't afford to store a full backup. We've had parity checks appended to the very same data to be protected for so long because it does protect from some common failure modes even if not from all. Dvdisaster goes one step beyond and gives the option to store the ECC file elsewhere. Definitely better than nothing and probably even with a very good cost/benefit ratio.
  - self_awareness 2y ago
    buran77 wrote: > You'd be comparing to an old optical disc. So almost anything is higher > reliability than that, starting with a newer optical disc and probably ending > at any cloud storage. Is "cloud" really being considered as a viable backup solution? I mean, some of my CDs are older than most of today's "cloud" servers and can be read just fine.
    - buran77 2y ago
      > some of my CDs are older than most of today's "cloud" servers and can be read just fine. And if you're happy being able to recover "some" of your data that's a perfectly legitimate option. Write CDs and DVDs, save to HDDs and SSDs, to the cloud, print on paper, or a combination of all. Only you know your data and its importance. But what you'll want to ask yourself though is how many CDs written since than can't be read just fine. I have books from before electricity was a thing and still am thoroughly convinced paper is not the best for long term preservation. My oldest cloud stored data that I can verify is a 25 year old email in my Yahoo mailbox. I have stamped CDs that survived that long, no written CD that I can think of.
      - self_awareness 2y ago
        Well, Yahoo is one of the few e-mail providers that survived the last 25 years. I had e-mails on multiple providers which don't exist since a long time. That brings another question, how do you even know that the cloud company will even exist in the next 25 years? I think Yahoo is now enforcing OAuth authorization. Maybe next step is to remove IMAP? How do you know that they're not planning it? The point is that it's impossible to try and predict how the cloud will look like in the future. One day we can end up receiving an e-mail that we need to download all data from the cloud because they're ending their operation. And the future of paper/CDs can be predicted easily. If anyone's worried about the lifespan of CDs, it's always possible to copy the collection once every few years. Also I think that paper could be a viable backup solution, at least for e.g. family photos -- print, encapsulate, hide in a dark and dry place.
        buran77 2y ago
        > how do you even know that the cloud company will even exist in the next 25 years You don't and trying to guess which technology or provider will still exist in 25 years is probably pointless. If you want to keep your data for extended periods of time (decades) the best strategy is to a) define what you want to protect against and don't go for "everything" or it will be prohibitively expensive and impractical, b) have at least one redundancy (an ECC file, a full copy, whatever helps you recover the data if one data set is affected), and c) ensure that the data sets cannot fail at the same time, statistically speaking, by using completely different methods. A CD will fail suddenly because you probably can't check it often enough. A major cloud provider doesn't disappear over night though. Keep the parity file in two completely unrelated clouds maybe even for free, or keep the whole data in highly compressed archives with recovery record in multiple clouds for a cost. Keep a full copy of your data on different HDDs with (different?) resilient file systems, and connect them periodically for a sanity check. Check periodically that you can still use the data, having an intact file isn't enough if you can no longer open that format used in 1993. There is no one size fits all and the is no set and forget "this backup will be guaranteed for the lifetime of the universe" answer. Choose what you protect against and choose how much you're willing to pay in time, effort, money. Then play whack-a-mole as some of your backups are no longer viable and you replace them to always have that +1 minimum resilience.
- poikroequ 2y ago
  Are hard drives such a terrible solution? Setup a RAID5 or RAID6 array and you could easily backup thousands of DVDs for less than $1000.
  - stavros 2y ago
    For around five years, sure.
    - poikroequ 2y ago
      I have hard drives which are 15 years old that still work just fine. Yeah, hard drives do fail, that's why you use RAID5 or RAID6. 8TB HDDs can be had for around $100 each. Four of these in RAID5 gives you 24TB. That's enough to backup 5000 DVDs (single layer) or 2800 dual layer DVDs.
      - stavros 2y ago
        I have a NAS with four drives, I had to replace all four of them 5-7 years later. You must be very lucky if your drives work 15 years later.
        buran77 2y ago
        stavros, to understand if any precautionary measure is effective you must first understand what you're protecting yourself against. Your line of argumentation is moving the goalposts a lot and punctuating each stop with "that's not good". Between the lines I take it you're aiming for "perfect" or at the very least "absolute best" protection. The first doesn't exist, the second is only interesting for only very few. If you're protecting your data from minor damage and decay of the DVD that stores it, having an ECC file stored elsewhere (a newer DVD, or HDD, or cloud) is the lowest effort and cost option but still a big step forward in resilience. It doesn't matter how long each medium lasts, all that matters is that they don't fail at the same time. Pick your HDDs from different manufacturers, models, or batches and the chance that all fail within the same short time period without external influence are exceptionally low. And when one does fail you replace it. You'll never get a one time perfect solution, you're just buying time to rebuild your resilience when you take a hit. If you're protecting against the complete destruction of the disk then an ECC file won't do, you'll need a full copy. Or multiple copies, and in multiple locations. Or on different mediums with very different reliability curves. You might want to protect against internal or external malicious actions, technical obsolescence of the medium or data format, vendor lock-in, or political instability. No one size fits all solution, certainly not a cost conscious one.

stavros 2y ago

What sort of higher-reliability options (than a Blu-ray) does a consumer have? I struggle to think of something that's not fairly niche, eg a magnetic tape drive.

buran77 2y ago
> higher-reliability options You'd be comparing to an old optical disc. So almost anything is higher reliability than that, starting with a newer optical disc and probably ending at any cloud storage. > does a consumer have This isn't limited just to your average consumer. DVDs are still in use in libraries and different archives which can't afford to store a full backup. We've had parity checks appended to the very same data to be protected for so long because it does protect from some common failure modes even if not from all. Dvdisaster goes one step beyond and gives the option to store the ECC file elsewhere. Definitely better than nothing and probably even with a very good cost/benefit ratio.
- self_awareness 2y ago
  buran77 wrote: > You'd be comparing to an old optical disc. So almost anything is higher > reliability than that, starting with a newer optical disc and probably ending > at any cloud storage. Is "cloud" really being considered as a viable backup solution? I mean, some of my CDs are older than most of today's "cloud" servers and can be read just fine.
  - buran77 2y ago
    > some of my CDs are older than most of today's "cloud" servers and can be read just fine. And if you're happy being able to recover "some" of your data that's a perfectly legitimate option. Write CDs and DVDs, save to HDDs and SSDs, to the cloud, print on paper, or a combination of all. Only you know your data and its importance. But what you'll want to ask yourself though is how many CDs written since than can't be read just fine. I have books from before electricity was a thing and still am thoroughly convinced paper is not the best for long term preservation. My oldest cloud stored data that I can verify is a 25 year old email in my Yahoo mailbox. I have stamped CDs that survived that long, no written CD that I can think of.
    - self_awareness 2y ago
      Well, Yahoo is one of the few e-mail providers that survived the last 25 years. I had e-mails on multiple providers which don't exist since a long time. That brings another question, how do you even know that the cloud company will even exist in the next 25 years? I think Yahoo is now enforcing OAuth authorization. Maybe next step is to remove IMAP? How do you know that they're not planning it? The point is that it's impossible to try and predict how the cloud will look like in the future. One day we can end up receiving an e-mail that we need to download all data from the cloud because they're ending their operation. And the future of paper/CDs can be predicted easily. If anyone's worried about the lifespan of CDs, it's always possible to copy the collection once every few years. Also I think that paper could be a viable backup solution, at least for e.g. family photos -- print, encapsulate, hide in a dark and dry place.
      - buran77 2y ago
        > how do you even know that the cloud company will even exist in the next 25 years You don't and trying to guess which technology or provider will still exist in 25 years is probably pointless. If you want to keep your data for extended periods of time (decades) the best strategy is to a) define what you want to protect against and don't go for "everything" or it will be prohibitively expensive and impractical, b) have at least one redundancy (an ECC file, a full copy, whatever helps you recover the data if one data set is affected), and c) ensure that the data sets cannot fail at the same time, statistically speaking, by using completely different methods. A CD will fail suddenly because you probably can't check it often enough. A major cloud provider doesn't disappear over night though. Keep the parity file in two completely unrelated clouds maybe even for free, or keep the whole data in highly compressed archives with recovery record in multiple clouds for a cost. Keep a full copy of your data on different HDDs with (different?) resilient file systems, and connect them periodically for a sanity check. Check periodically that you can still use the data, having an intact file isn't enough if you can no longer open that format used in 1993. There is no one size fits all and the is no set and forget "this backup will be guaranteed for the lifetime of the universe" answer. Choose what you protect against and choose how much you're willing to pay in time, effort, money. Then play whack-a-mole as some of your backups are no longer viable and you replace them to always have that +1 minimum resilience.
poikroequ 2y ago
Are hard drives such a terrible solution? Setup a RAID5 or RAID6 array and you could easily backup thousands of DVDs for less than $1000.
- stavros 2y ago
  For around five years, sure.
  - poikroequ 2y ago
    I have hard drives which are 15 years old that still work just fine. Yeah, hard drives do fail, that's why you use RAID5 or RAID6. 8TB HDDs can be had for around $100 each. Four of these in RAID5 gives you 24TB. That's enough to backup 5000 DVDs (single layer) or 2800 dual layer DVDs.
    - stavros 2y ago
      I have a NAS with four drives, I had to replace all four of them 5-7 years later. You must be very lucky if your drives work 15 years later.
      - buran77 2y ago
        stavros, to understand if any precautionary measure is effective you must first understand what you're protecting yourself against. Your line of argumentation is moving the goalposts a lot and punctuating each stop with "that's not good". Between the lines I take it you're aiming for "perfect" or at the very least "absolute best" protection. The first doesn't exist, the second is only interesting for only very few. If you're protecting your data from minor damage and decay of the DVD that stores it, having an ECC file stored elsewhere (a newer DVD, or HDD, or cloud) is the lowest effort and cost option but still a big step forward in resilience. It doesn't matter how long each medium lasts, all that matters is that they don't fail at the same time. Pick your HDDs from different manufacturers, models, or batches and the chance that all fail within the same short time period without external influence are exceptionally low. And when one does fail you replace it. You'll never get a one time perfect solution, you're just buying time to rebuild your resilience when you take a hit. If you're protecting against the complete destruction of the disk then an ECC file won't do, you'll need a full copy. Or multiple copies, and in multiple locations. Or on different mediums with very different reliability curves. You might want to protect against internal or external malicious actions, technical obsolescence of the medium or data format, vendor lock-in, or political instability. No one size fits all solution, certainly not a cost conscious one.