3 ms·
I would expect that filesystems do not integrate conventional FEC (e.g. block codes), because FEC is poorly suited to the kind of errors that affect filesystems
by variaga 6y ago
I would expect that filesystems do not integrate conventional FEC (e.g. block codes), because FEC is poorly suited to the kind of errors that affect filesystems.
FEC works against errors that have a predictable expected error rate, where errors are evenly distributed among the data. This applies to block codes (where FEC will have a maximum decodable error rate per block) and convolutional codes which don't have 'blocks', but still have a maximum local error density they can reliably correct. For media where burst errors are a possibility, FEC is almost always paired with interleaving to increase the likelihood hor a ~uniform distribution of errors.
FEC works pretty well when the problem is "I wrote some data to disk that I can't read back correctly, but I can read the other data back just fine". At the media level, hard drives have used FEC to protect individual bits/sectors for a long time. Likewise at the disk level, RAID 4/5/6 do use FEC to protect against individual disk failures.
Errors experienced at a filesystem level are not usually due to 'random' corruption of the data, but to invalid operations on the data structure e.g. a crash while flushing the data to disk which leaves the filesystem in an inconsistent state. In other words, the filesystem-specific errors are more likely to be "I wrote the wrong data to disk" or "I only wrote some of the data I meant to".
FEC will not defend against that well, because any FEC must be written to the disk along with the data to prevent the FEC+data from getting into an inconsistent state. For example, the 'RAID 5 write hole' (which also happens with RAID 1 and RAID 6) exists because physical disks will not commit data to the disk at exactly the same time even when the write commands are issued at the same time (which mostly won't happen anyway, since write commands tend to be serialized on your SCSI/Fiberchannel/whatever interface to multiple drives). That means there exists a time where the data has been updated but the parity has not (or vice versa), and if a drive fails at that time, the RAID array may not be cleanly recoverable.
So for practical reasons, if you write the wrong data to disk, you'll probably write the wrong FEC too, and FEC won't help recover much. If your allocation table gets corrupted and you write part of one file on top of another, you'll be overwriting the FEC too. For conventional FEC.
BUT... error correction isn't just "conventional FEC". Error correction in general is any time you add redundancy to data to make it more tolerant to errors. In the filesystem space, this happens, but it isn't usually done with parity matricies or convolutional state machines. Instead, redundancy is added by things like log-structured file systems. The 'write what you plan to do, then write the actual data, then write that you successfully did it (in that order)' behavior of log structured file systems does add redundancy and does make error recovery more possible.
In summary, "error correction" in general is used at several levels for file storage:
- block FEC at the bit level
- block FEC at the drive level if RAID is used
- robust CRC error detect + automatic repeat request over your PCIe/NVMe link or over a network connection
- Redundant metadata in your filesystem journal
but the key is to use the right kind of error correction for the expected error conditions.
- emilfihlman 6y agoI suggest you read a bit more about reliable data transmission and FEC. Burst error FECs absolutely available and used (mostly FEC + interleaving in this order).