3 ms·

Intel's Storage Acceleration Library

30 points by sharva 10y ago

jerrysievert 10y ago
it would be very helpful if they included a description as to what it is this library does.
byefruit 10y ago
https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version https://01.org/intel%C2%AE-storage-acceleration-library-open... has more info
notmyname 10y ago
I've used ISA-L as the backend implementation for erasure codes in OpenStack Swift. There are two versions of ISA-L. The open-source one includes some optimized erasure code libraries. The other version is free (as in beer) AFAICT but it requires signing a license agreement with Intel. This version also includes some other optimized libraries for hashing, crypto, and stuff like that. https://github.com/openstack/liberasurecode https://github.com/openstack/liberasurecode is a plugable C library that you can use for erasure coding. It supports some simple EC algorithms itself and works with jerasure and ISA-L. https://github.com/openstack/pyeclib https://github.com/openstack/pyeclib is a Python wrapper for liberasurecode.
dragontamer 10y ago
This appears to be an implementation of Reed-solomon error correction code... at the assembly level. (!! handwritten assembly? I don't see that too often nowadays) https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor... Which is basically the mathematics behind RAID-5, RAID-6, and even hard-drive / SSD sector-level error correction. Parchive is an older open-source project that implements the same concept: https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
- beagle3 10y ago
  RAID-5 uses simply parity, RAID-6 uses something slightly more complicated, but they are not close in complexity (or capabilities) of Reed-Solomon codes -- although, if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.
  - dragontamer 10y ago
    You're right about Raid5. My understanding is that Raid-6 is Reed-solomon however: https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 > Several methods, including dual check data computations (parity and Reed-Solomon), orthogonal dual parity check data and diagonal parity, have been used to implement RAID Level 6 > if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes. Reed-Solomon codes are generalized for any number of drives with any number of failures. There's a bitwise-efficiency thing with powers-of-two as well. I forgot the details, but I did the math back in college. I'll have to find my notes on this subject... but yeah, the matrix works for virtually everything. I forgot whether or not Reed-Solomon becomes just a simple xor-parity check in the case of 1-failure, but it wouldn't surprise me if it were related.

30 points by sharva 10y ago

it would be very helpful if they included a description as to what it is this library does.

https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version https://01.org/intel%C2%AE-storage-acceleration-library-open... has more info

notmyname 10y ago

I've used ISA-L as the backend implementation for erasure codes in OpenStack Swift. There are two versions of ISA-L. The open-source one includes some optimized erasure code libraries. The other version is free (as in beer) AFAICT but it requires signing a license agreement with Intel. This version also includes some other optimized libraries for hashing, crypto, and stuff like that. https://github.com/openstack/liberasurecode https://github.com/openstack/liberasurecode is a plugable C library that you can use for erasure coding. It supports some simple EC algorithms itself and works with jerasure and ISA-L. https://github.com/openstack/pyeclib https://github.com/openstack/pyeclib is a Python wrapper for liberasurecode.

dragontamer 10y ago

This appears to be an implementation of Reed-solomon error correction code... at the assembly level. (!! handwritten assembly? I don't see that too often nowadays) https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor... Which is basically the mathematics behind RAID-5, RAID-6, and even hard-drive / SSD sector-level error correction. Parchive is an older open-source project that implements the same concept: https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive

beagle3 10y ago
RAID-5 uses simply parity, RAID-6 uses something slightly more complicated, but they are not close in complexity (or capabilities) of Reed-Solomon codes -- although, if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.
- dragontamer 10y ago
  You're right about Raid5. My understanding is that Raid-6 is Reed-solomon however: https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 > Several methods, including dual check data computations (parity and Reed-Solomon), orthogonal dual parity check data and diagonal parity, have been used to implement RAID Level 6 > if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes. Reed-Solomon codes are generalized for any number of drives with any number of failures. There's a bitwise-efficiency thing with powers-of-two as well. I forgot the details, but I did the math back in college. I'll have to find my notes on this subject... but yeah, the matrix works for virtually everything. I forgot whether or not Reed-Solomon becomes just a simple xor-parity check in the case of 1-failure, but it wouldn't surprise me if it were related.