4 ms·
Good link! I've both forwarded it on, and will study it when I have some time later.
by brianwski 8y ago
Good link! I've both forwarded it on, and will study it when I have some time later.
- jorangreef 8y agoHi Brian, thanks for open-sourcing Backblaze's JavaReedSolomon, which is really well-written. A few months ago I ran into an issue with Reed Solomon coding throughput not saturating the write throughput of 16 drives, and wrote a new Reed Solomon module based on Cauchy matrices: https://github.com/ronomon/reed-solomon https://github.com/ronomon/reed-solomon The Cauchy matrices remove the need for a table lookup to do the Galois multiply, replacing it with pure XOR. Together with other optimizations, this gives nearly 3x-5x more coding throughput for the same (17,3) parameters, assuming you're still using your open-sourced JavaReedSolomon in production. I don't know if Reed Solomon coding throughput is a factor in your rebuild times?
- kdkeyser 8y agoThere is also Intel's ISA-L: https://github.com/01org/isa-l https://github.com/01org/isa-l It contains optimized Galois Field multiplication, resulting in Reed Solomon (both Vandermonde as well as Cauchy) at multiple GB/s on a modern x86 CPU. Still, I doubt that the Reed Solomon coding speed is the limiting factor in their rebuild time. There is a mention of a 6-day duration, so even with a very slow Reed Solomon implementation ( ~ 100 MB/s) that should not be a bottleneck for a 10 TB drive rebuild (assuming a distributed rebuild approach, not a traditional RAID style rebuild).