4 ms·

Can you link to any source giving details on this? I don't know of any mechanism that can protect individual bits from flipping without dedicating a serious amo

Can you link to any source giving details on this? I don't know of any mechanism that can protect individual bits from flipping without dedicating a serious amount of space to redundancy. And it doesn't align with my personal experience either: I had a bit flip in a Jpeg image this year. Fortunately I was able to restore it from backup.

vbezhenar 7y ago
It's pretty common knowledge. You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips. Also disks use checksums which require even less storage. Your situation is likely happened because of bad RAM.
- bscphil 7y ago
  > Your situation is likely happened because of bad RAM. I don't see how that could have happened, since the image wasn't ever written to. It should have been the same file on disk as the backup, since I hadn't touched it since backing up the file, but it wasn't. > You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips The thing about parity is that for it to be useful in a data recovery scheme, you have to know what bit got flipped. That works for RAID of course, because you usually know which disk is bad and so when you replace it you can work out the missing bits using the parity disk, but I don't think it could work for an individual hard drive, since in general you don't know which bit flipped. And the thing about checksums is that they can tell you if your data has become corrupt, but they can't be used to fix things behind the scenes.
pmarreck 7y ago
Turbo codes don't add a lot of overhead if you just want to protect against 1 bit flip per byte or so via FEC (forward error-correction), they're close to the Shannon information entropy limit in efficiency but I don't know of any open source implementations, and implementation details such as "I want to protect against at most 2 bits per byte" or "I want to protect at most 5 bytes per kilobyte" directly matter in the implementation. I've been wanting to try to understand them to see if I could implement something myself for a while now
- planteen 7y ago
  Turbo codes are typically soft decision codes, which don't make a ton of sense at the filesystem level, since there has already been a hard decision made. They are useful in storage at the read channel level, as in processing the analog stream from the head on the hard drive. Reed-Solomon is often used in storage because it is an optimal erasure code - e.g., I know this block is missing or corrupt, correct it.
  - bscphil 7y ago
    > Reed-Solomon is often used in storage because it is an optimal erasure code I was initially thinking OP might be talking about something like this, but I think I would have heard if all new hard drives had it built into their firmware to store error correcting codes alongside the real data and automatically fix bit flips, since I suspect that would have pretty serious performance impacts in certain cases and people would be complaining.

vbezhenar 7y ago

It's pretty common knowledge. You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips. Also disks use checksums which require even less storage. Your situation is likely happened because of bad RAM.

bscphil 7y ago
> Your situation is likely happened because of bad RAM. I don't see how that could have happened, since the image wasn't ever written to. It should have been the same file on disk as the backup, since I hadn't touched it since backing up the file, but it wasn't. > You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips The thing about parity is that for it to be useful in a data recovery scheme, you have to know what bit got flipped. That works for RAID of course, because you usually know which disk is bad and so when you replace it you can work out the missing bits using the parity disk, but I don't think it could work for an individual hard drive, since in general you don't know which bit flipped. And the thing about checksums is that they can tell you if your data has become corrupt, but they can't be used to fix things behind the scenes.

pmarreck 7y ago

Turbo codes don't add a lot of overhead if you just want to protect against 1 bit flip per byte or so via FEC (forward error-correction), they're close to the Shannon information entropy limit in efficiency but I don't know of any open source implementations, and implementation details such as "I want to protect against at most 2 bits per byte" or "I want to protect at most 5 bytes per kilobyte" directly matter in the implementation. I've been wanting to try to understand them to see if I could implement something myself for a while now

planteen 7y ago
Turbo codes are typically soft decision codes, which don't make a ton of sense at the filesystem level, since there has already been a hard decision made. They are useful in storage at the read channel level, as in processing the analog stream from the head on the hard drive. Reed-Solomon is often used in storage because it is an optimal erasure code - e.g., I know this block is missing or corrupt, correct it.
- bscphil 7y ago
  > Reed-Solomon is often used in storage because it is an optimal erasure code I was initially thinking OP might be talking about something like this, but I think I would have heard if all new hard drives had it built into their firmware to store error correcting codes alongside the real data and automatically fix bit flips, since I suspect that would have pretty serious performance impacts in certain cases and people would be complaining.