5 ms·
The next level of efficiency is using nested erasure codes. The outer code can be across regions/zones/machines/disks while the inner code is across chunks of
by benlivengood 3y ago
The next level of efficiency is using nested erasure codes. The outer code can be across regions/zones/machines/disks while the inner code is across chunks of a stripe. Chunk unavailability is fast to correct with an extra outer chunk and bit rot or corruption can be fixed by the inner code without an extra fetch. In the fast path only data chunks need to be fetched.
- dragontamer 3y agoThat's just LDPC but not as extreme. If you're going to "nest" erasure codes, might as well make them XOR-based (fastest operation on modern CPUs and/or hardware), calculate a randomization scheme that has very-very high probability (99.9%+) of fixing errors, and other such benefits. To provide assurances that you have enough LDPC, you then run LDPC on your LDPC-check bits. Then you run LDPC on your LDPC-LDPC-check bits. Then you run LDPC on your LPDC-LDPC-LDPC-check bits, until you've got the target probability / chance of fixing errors that you desire. -------- LDPC's brilliance is that XOR-as-erasure-code is exceptionally fast, and this "repeated hierarchy" of error-correction codes leads to high-probabilities (99.9999%+) of successful correction of erasures.