4 ms·

Anyone know of a comparison of similar tools. I've tinkered with this one: https://github.com/klauspost/reedsolomon/ https://github.com/klauspost/reedsolomon/

by sliken 10y ago

Anyone know of a comparison of similar tools. I've tinkered with this one: https://github.com/klauspost/reedsolomon/ https://github.com/klauspost/reedsolomon/ I believe it's compatible with the Java implementation used by Backblaze and others.

jorangreef 10y ago
I don't know of any cross-library comparison benchmark. Also, if a library does ship with an internal benchmark, it usually uses a different number of data/parity shards and different shard size, compared to other library benchmarks, which makes comparison difficult. I wrote an MIT-licensed ReedSolomon module for Node [1], with native C++ multi-core throughput. The internal benchmark (node benchmark.js) provides latency and throughput measurements for data=10, parity=4 for a variety of shard sizes, and the number of data/parity shards can be adjusted for comparison. It's not yet using optimized assembler or SIMD like Klaus Post's implementation, but it is compatible and also based on Backblaze's implementation, as well as including a pure Javascript implementation for browser use. [1] https://github.com/jorangreef/reed-solomon https://github.com/jorangreef/reed-solomon
nickcw 10y ago
The one you linked to is excellent. It has hand crafted SSE3 assembler for enormous speed. It is a bit difficult to compare the benchmarks from the one in the original posting to Klaus's as they aren't done at the same (N,K) parameters though.
deleted 10y ago
[deleted]
vu3rdd 10y ago
One of the authors of Tahoe-LAFS and the original creator of zfec, Zooko, co-authored a paper on the performance of such codes. http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf Luigi Rizzo is the original author of the fec C library. I co-maintain zfec (and the Debian package of it) along with Zooko. A new release of zfec is planned to be out soonish. edit: removing the <> delimiters around the URL. Sorry!
- zokier 10y ago
  HN sadly does not support <> URL delimiters, so your link ends up 404. Here is it without delimiters: http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf
  - vu3rdd 10y ago
    Thanks. Updated the link.
ironMann 10y ago
Papers usually compare against jerasure: http://jerasure.org/jerasure/jerasure http://jerasure.org/jerasure/jerasure
nicolast 10y ago
I ported Klaus' library to Haskell, and redid the whole SIMD implementation (see below for the extras this brings): https://github.com/NicolasT/reedsolomon/ https://github.com/NicolasT/reedsolomon/ In my library, the SSSE3 code also has AVX and AVX2 equivalents, providing higher throughput. There are also implementations with specific optimized instructions for ARM NEON, ARM64, PPC64/AltiVec and Power8. All of this in C (with intrinsics), not ASM, so making use of your C compiler's instruction scheduling, loop unrolling,... See https://github.com/NicolasT/reedsolomon/blob/master/cbits/reedsolomon.c https://github.com/NicolasT/reedsolomon/blob/master/cbits/re...

by sliken 10y ago

jorangreef 10y ago

I don't know of any cross-library comparison benchmark. Also, if a library does ship with an internal benchmark, it usually uses a different number of data/parity shards and different shard size, compared to other library benchmarks, which makes comparison difficult. I wrote an MIT-licensed ReedSolomon module for Node [1], with native C++ multi-core throughput. The internal benchmark (node benchmark.js) provides latency and throughput measurements for data=10, parity=4 for a variety of shard sizes, and the number of data/parity shards can be adjusted for comparison. It's not yet using optimized assembler or SIMD like Klaus Post's implementation, but it is compatible and also based on Backblaze's implementation, as well as including a pure Javascript implementation for browser use. [1] https://github.com/jorangreef/reed-solomon https://github.com/jorangreef/reed-solomon

nickcw 10y ago

The one you linked to is excellent. It has hand crafted SSE3 assembler for enormous speed. It is a bit difficult to compare the benchmarks from the one in the original posting to Klaus's as they aren't done at the same (N,K) parameters though.

deleted 10y ago

[deleted]

vu3rdd 10y ago

One of the authors of Tahoe-LAFS and the original creator of zfec, Zooko, co-authored a paper on the performance of such codes. http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf Luigi Rizzo is the original author of the fec C library. I co-maintain zfec (and the Debian package of it) along with Zooko. A new release of zfec is planned to be out soonish. edit: removing the <> delimiters around the URL. Sorry!

zokier 10y ago
HN sadly does not support <> URL delimiters, so your link ends up 404. Here is it without delimiters: http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf
- vu3rdd 10y ago
  Thanks. Updated the link.

ironMann 10y ago

Papers usually compare against jerasure: http://jerasure.org/jerasure/jerasure http://jerasure.org/jerasure/jerasure

nicolast 10y ago

I ported Klaus' library to Haskell, and redid the whole SIMD implementation (see below for the extras this brings): https://github.com/NicolasT/reedsolomon/ https://github.com/NicolasT/reedsolomon/ In my library, the SSSE3 code also has AVX and AVX2 equivalents, providing higher throughput. There are also implementations with specific optimized instructions for ARM NEON, ARM64, PPC64/AltiVec and Power8. All of this in C (with intrinsics), not ASM, so making use of your C compiler's instruction scheduling, loop unrolling,... See https://github.com/NicolasT/reedsolomon/blob/master/cbits/reedsolomon.c https://github.com/NicolasT/reedsolomon/blob/master/cbits/re...