3 ms·

Another tool that is the de-facto standard on usenet for erasure coding is .par2 https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive

by 0x0 10y ago

djsumdog 10y ago
I was going to ask if this was like PAR.
gwern 10y ago
I'm not sure PAR2 really counts as erasure coding rather than a more general FEC. I've tried zfec in the past but it's not really set up for backup uses such as creating an archive to burn to a set of BD disks. For backups, what you want to generate additional files, par2 style, which can be combined with the regular archive files (eg like in duplicity, GPG-encrypted tarballs) to repair any bitrot or lost fractions of those archives. But with zfec, as I understood it, it only works if you turn them entirely into erasure-coded shares and assume you'll lose entire shares and wind up using m-of-n shares in recovery/restoration (as opposed to PAR2 which will handle losing individual archive files but also arbitrary corruption in any of those archive files, as long as the total does not exceed the redundancy %). So you would have to do something like create a single giant archive file, and if you wanted the equivalent of 50% redundancy, you'd use zfec to turn it into '50-of-100' shares and backup the 100 shares (or do 500-of-1000 etc). Then any shares which aren't bit-identical to get tossed out at restoration and you hope you have enough shares left to reconstruct the original giant archive file. But you might not - your shares might be mostly good but missing a few chunks, which is something a PAR2 setup could cope with (if the chunks are in the archive files, they get repaired normally, and if they're in the PAR2 redundancy, they don't matter). This is fine in the Taho-LAFS or datacenter setup where you assume your storage nodes will store perfectly or fail entirely (since given any live errors, you can just scrub the entire machine and rebuild an additional copy from the erasure-coded shares). Not so much in a backup setting. Ultimately, while zfec is way faster than par2, I found this setup squirrely enough that I wasn't convinced to switch.

Another tool that is the de-facto standard on usenet for erasure coding is .par2 https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive

by 0x0 10y ago

Another tool that is the de-facto standard on usenet for erasure coding is .par2 https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive

djsumdog 10y ago

I was going to ask if this was like PAR.

gwern 10y ago

I'm not sure PAR2 really counts as erasure coding rather than a more general FEC. I've tried zfec in the past but it's not really set up for backup uses such as creating an archive to burn to a set of BD disks. For backups, what you want to generate additional files, par2 style, which can be combined with the regular archive files (eg like in duplicity, GPG-encrypted tarballs) to repair any bitrot or lost fractions of those archives. But with zfec, as I understood it, it only works if you turn them entirely into erasure-coded shares and assume you'll lose entire shares and wind up using m-of-n shares in recovery/restoration (as opposed to PAR2 which will handle losing individual archive files but also arbitrary corruption in any of those archive files, as long as the total does not exceed the redundancy %). So you would have to do something like create a single giant archive file, and if you wanted the equivalent of 50% redundancy, you'd use zfec to turn it into '50-of-100' shares and backup the 100 shares (or do 500-of-1000 etc). Then any shares which aren't bit-identical to get tossed out at restoration and you hope you have enough shares left to reconstruct the original giant archive file. But you might not - your shares might be mostly good but missing a few chunks, which is something a PAR2 setup could cope with (if the chunks are in the archive files, they get repaired normally, and if they're in the PAR2 redundancy, they don't matter). This is fine in the Taho-LAFS or datacenter setup where you assume your storage nodes will store perfectly or fail entirely (since given any live errors, you can just scrub the entire machine and rebuild an additional copy from the erasure-coded shares). Not so much in a backup setting. Ultimately, while zfec is way faster than par2, I found this setup squirrely enough that I wasn't convinced to switch.