3 ms·

Is there any system for error recovery in userspace? Something like, I’ve got directory DATA, i want to create a recovery file for all the contents in such dire

Is there any system for error recovery in userspace? Something like, I’ve got directory DATA, i want to create a recovery file for all the contents in such directory. Ideally I’d like to be able to tune the size of the recovery, larger recovery = recover more errors

auxym 6y ago
We used to use this when creating backups on CD-Rs and later DVDs, before hard drives became cheap... https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
- bestham 6y ago
  I used a program named Dvdisaster for the same use case. You could use it I two modes: (1) One disc with additional out of band reed-solomon ECC. Create an iso that are smaller by a certain percentage than the medium you want to burn it to. Dvdisaster will create a new ISO that uses the full capacity of the medium and fill the remaining space with parity-information that protects the whole file system and all user files. (2) Put your parity information on the next disc you burn. A parity file is crated for the first disc you burn, this is later added to the file system of the second disc you burn. Repeat indefinitely. You always have the last yet-to-be-burned parity file on your computer and can thus recover bit-errors in every disc you have burned.
- dylan604 6y ago
  We used to use this in Newsgroup days. Large files got broken into smaller chunks (maybe floppy disk sized??) in the RAR format. You could recover missing files if they expired before you get them downloaded on your dial-up modem or your particular newsgroup service wasn't reliable enough.
dmayle 6y ago
What you're looking for is called Parchive. It uses Reed-Solomon coding in order to generate redundant files that can be used to recover the original. Look for PAR2 utilities (the current major version of the format). It has multiple implementations, open source and closed, across platforms and architectures. https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
- brnt 6y ago
  I wrote a gui for it, tot help deal with many files being in various states: https://github.com/brenthuisman/par2deep https://github.com/brenthuisman/par2deep
- thomasmg 6y ago
  Another option is to use an IBLT (Inverted Bloom Lookup Table). it is easy to implement. I recently wrote a file repair tool that uses it: https://github.com/thomasmueller/tinyStats/blob/master/src/main/java/org/tinyStats/tools/FileRepair.java https://github.com/thomasmueller/tinyStats/blob/master/src/m... (not production ready)
sliken 6y ago
This isn't a turn key utility, but it's relatively easy to call from Go. It allows you to set the number of data shards and parity shards, it's pretty fast. It's at https://github.com/klauspost/reedsolomon Seems like a pretty common solution, I suspect there's other libraries for popular language/platforms.

auxym 6y ago

We used to use this when creating backups on CD-Rs and later DVDs, before hard drives became cheap... https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive

bestham 6y ago
I used a program named Dvdisaster for the same use case. You could use it I two modes: (1) One disc with additional out of band reed-solomon ECC. Create an iso that are smaller by a certain percentage than the medium you want to burn it to. Dvdisaster will create a new ISO that uses the full capacity of the medium and fill the remaining space with parity-information that protects the whole file system and all user files. (2) Put your parity information on the next disc you burn. A parity file is crated for the first disc you burn, this is later added to the file system of the second disc you burn. Repeat indefinitely. You always have the last yet-to-be-burned parity file on your computer and can thus recover bit-errors in every disc you have burned.
dylan604 6y ago
We used to use this in Newsgroup days. Large files got broken into smaller chunks (maybe floppy disk sized??) in the RAR format. You could recover missing files if they expired before you get them downloaded on your dial-up modem or your particular newsgroup service wasn't reliable enough.

dmayle 6y ago

What you're looking for is called Parchive. It uses Reed-Solomon coding in order to generate redundant files that can be used to recover the original. Look for PAR2 utilities (the current major version of the format). It has multiple implementations, open source and closed, across platforms and architectures. https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive

brnt 6y ago
I wrote a gui for it, tot help deal with many files being in various states: https://github.com/brenthuisman/par2deep https://github.com/brenthuisman/par2deep
thomasmg 6y ago
Another option is to use an IBLT (Inverted Bloom Lookup Table). it is easy to implement. I recently wrote a file repair tool that uses it: https://github.com/thomasmueller/tinyStats/blob/master/src/main/java/org/tinyStats/tools/FileRepair.java https://github.com/thomasmueller/tinyStats/blob/master/src/m... (not production ready)

sliken 6y ago

This isn't a turn key utility, but it's relatively easy to call from Go. It allows you to set the number of data shards and parity shards, it's pretty fast. It's at https://github.com/klauspost/reedsolomon Seems like a pretty common solution, I suspect there's other libraries for popular language/platforms.