5 ms·

What prevents, or tries to prevent, bit rot in standard CPU hard drives/flash drives? Is there some built-in forward error correction at some layer? If not, how

What prevents, or tries to prevent, bit rot in standard CPU hard drives/flash drives? Is there some built-in forward error correction at some layer? If not, how could one add some?

Tomte 7y ago
Nothing. There's par2 (and its offsprings MultiPar and QuickPar), there's rsbep, there's pyFileFixity. All have their problems why you probably won't "shield" your files with them (I don't, although I worry).
- planteen 7y ago
  Curious - what are your complaints on these programs? My main one is that they are slow to create archives and rsbep only does error decoding, not erasure. I'm working on an alternative tool that is much faster, but curious if you have any other feedback.
wazoox 7y ago
Most SSDs only guarantee your data for 90 days when off. Even high grade, SLC, professional SSDs. Hard drives fare better, but your best bet for decade-long conservation remains tape.
- ByThyGrace 7y ago
  Is anyone interested in amateur digital archiving just destined to losing it all in the long run? You'd think there'd be a market for long life HDDs. Edit: Yes, you could just replace old hardware with new hardware, but it creates so much waste...
  - Multicomp 7y ago
    That's where hopefully Microsoft project silica will make it permanent for all practical purposes digital data storage available. It apparently uses a similar basic process as those mall kiosks that take a volumetric picture of your face and then etch it into a cube of glass? But in this case, they just write the data to the side of a small piece of glass over the course of days and it's basically permanent for all eternity unless something breaks it.
- everdrive 7y ago
  Wait, what. An SSD won't hold your data for more than 90 days if it's powered off?
  - wazoox 7y ago
    It will probably hold it much longer. However it's not guaranteed to do any better than that.
    - pricechild 7y ago
      I don't think you'd have much luck getting anything back from that "guarantee"... so is it a guarantee?
- wtallis 7y ago
  That 90 days figure applies only to SSDs that have worn out their entire rated write endurance, and only to enterprise/datacenter SSDs. For client/consumer SSDs, the standard for unpowered data retention at the end of life is one year (at 30°C, compared to 40°C for enterprise drives). That longer retention requirement is one contributing factor to consumer SSDs being rated with lower write endurance. In practice, hardly anyone uses up the entire write endurance of their SSD, especially not if they're using it to store backups, archives, or other relatively static data. Flash memory that isn't worn out has much longer retention times.
vbezhenar 7y ago
Yes, all modern disks use error correcting. Wrong bit reads happen frequently in HDD and corrected all the time. Bits can't rot in HDD or SSD outside of firmware bugs or transmission errors. You'll get sector read error instead of wrong data.
- bscphil 7y ago
  Can you link to any source giving details on this? I don't know of any mechanism that can protect individual bits from flipping without dedicating a serious amount of space to redundancy. And it doesn't align with my personal experience either: I had a bit flip in a Jpeg image this year. Fortunately I was able to restore it from backup.
  - vbezhenar 7y ago
    It's pretty common knowledge. You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips. Also disks use checksums which require even less storage. Your situation is likely happened because of bad RAM.
    - bscphil 7y ago
      > Your situation is likely happened because of bad RAM. I don't see how that could have happened, since the image wasn't ever written to. It should have been the same file on disk as the backup, since I hadn't touched it since backing up the file, but it wasn't. > You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips The thing about parity is that for it to be useful in a data recovery scheme, you have to know what bit got flipped. That works for RAID of course, because you usually know which disk is bad and so when you replace it you can work out the missing bits using the parity disk, but I don't think it could work for an individual hard drive, since in general you don't know which bit flipped. And the thing about checksums is that they can tell you if your data has become corrupt, but they can't be used to fix things behind the scenes.
  - pmarreck 7y ago

What prevents, or tries to prevent, bit rot in standard CPU hard drives/flash drives? Is there some built-in forward error correction at some layer? If not, how could one add some?

Tomte 7y ago

Nothing. There's par2 (and its offsprings MultiPar and QuickPar), there's rsbep, there's pyFileFixity. All have their problems why you probably won't "shield" your files with them (I don't, although I worry).

planteen 7y ago
Curious - what are your complaints on these programs? My main one is that they are slow to create archives and rsbep only does error decoding, not erasure. I'm working on an alternative tool that is much faster, but curious if you have any other feedback.

wazoox 7y ago

Most SSDs only guarantee your data for 90 days when off. Even high grade, SLC, professional SSDs. Hard drives fare better, but your best bet for decade-long conservation remains tape.

ByThyGrace 7y ago
Is anyone interested in amateur digital archiving just destined to losing it all in the long run? You'd think there'd be a market for long life HDDs. Edit: Yes, you could just replace old hardware with new hardware, but it creates so much waste...
- Multicomp 7y ago
  That's where hopefully Microsoft project silica will make it permanent for all practical purposes digital data storage available. It apparently uses a similar basic process as those mall kiosks that take a volumetric picture of your face and then etch it into a cube of glass? But in this case, they just write the data to the side of a small piece of glass over the course of days and it's basically permanent for all eternity unless something breaks it.
everdrive 7y ago
Wait, what. An SSD won't hold your data for more than 90 days if it's powered off?
- wazoox 7y ago
  It will probably hold it much longer. However it's not guaranteed to do any better than that.
  - pricechild 7y ago
    I don't think you'd have much luck getting anything back from that "guarantee"... so is it a guarantee?
wtallis 7y ago
That 90 days figure applies only to SSDs that have worn out their entire rated write endurance, and only to enterprise/datacenter SSDs. For client/consumer SSDs, the standard for unpowered data retention at the end of life is one year (at 30°C, compared to 40°C for enterprise drives). That longer retention requirement is one contributing factor to consumer SSDs being rated with lower write endurance. In practice, hardly anyone uses up the entire write endurance of their SSD, especially not if they're using it to store backups, archives, or other relatively static data. Flash memory that isn't worn out has much longer retention times.

vbezhenar 7y ago

Yes, all modern disks use error correcting. Wrong bit reads happen frequently in HDD and corrected all the time. Bits can't rot in HDD or SSD outside of firmware bugs or transmission errors. You'll get sector read error instead of wrong data.

bscphil 7y ago
Can you link to any source giving details on this? I don't know of any mechanism that can protect individual bits from flipping without dedicating a serious amount of space to redundancy. And it doesn't align with my personal experience either: I had a bit flip in a Jpeg image this year. Fortunately I was able to restore it from backup.
- vbezhenar 7y ago
  It's pretty common knowledge. You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips. Also disks use checksums which require even less storage. Your situation is likely happened because of bad RAM.
  - bscphil 7y ago
    > Your situation is likely happened because of bad RAM. I don't see how that could have happened, since the image wasn't ever written to. It should have been the same file on disk as the backup, since I hadn't touched it since backing up the file, but it wasn't. > You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips The thing about parity is that for it to be useful in a data recovery scheme, you have to know what bit got flipped. That works for RAID of course, because you usually know which disk is bad and so when you replace it you can work out the missing bits using the parity disk, but I don't think it could work for an individual hard drive, since in general you don't know which bit flipped. And the thing about checksums is that they can tell you if your data has become corrupt, but they can't be used to fix things behind the scenes.
- pmarreck 7y ago