4 ms·
If you want to talk about low overhead redundancy schemes based on Reed Solomon coding, let me know, we used them to halve disk space requirements in QFS as opp
by paulsutter 12y ago
If you want to talk about low overhead redundancy schemes based on Reed Solomon coding, let me know, we used them to halve disk space requirements in QFS as opposed to HDFS (with no loss in reliability), and it could be stretched arbitrarily far to get at least some redundancy with an arbitrarily small overhead.
- db48x 12y agoWe opted not to do any fancy encoding of the data, even though it would make it easier to store (we would need fewer volunteers). Instead we store individual files whole and unencoded on each volunteer's machine, allowing the volunteer to make use of them as they see fit; no one has to be able to contact other volunteers in order to get useful data. Of course, since each volunteer's client is downloading files at random, you may not end up with the exact file you wanted (to pick an example at random, perhaps you wanted the fifth reel of the soundex index of the 1880 US Census for the state of Tennessee, but you wanted a collection of Lois Armstrong records from the 1920s instead). This turns out not to be a problem, however, as anyone can use git annex to manually pull down any file in the backup that they want. It gets counted as a copy by the backup system along with all the rest, and increases the redundancy of the backup.