3 ms·

Quite the opposite (it's like a RAID storage system), because of the error correcting scheme, you need a lot less storage to provide a good level of redundancy.

Quite the opposite (it's like a RAID storage system), because of the error correcting scheme, you need a lot less storage to provide a good level of redundancy. You only need to store pieces of size 1/N * file size, and as long as you can get N of those pieces you can reconstruct the file. To guarantee a high level of availability, you only need that there are more than N hosts up containing distinct pieces. So typically you will use (~3 * size of file) of storage on your network, instead of (number of seeds * size of file) bits, and still provide with the same level of service as if number of seeds where seeding the whole file. So each node is able to store a lot more pieces of other files.

eximius 8y ago
I might need to look up how they work because that's basically what my next question is: how do the chunks overlap and how does that affect the probability of a file being unrecoverable? If I have a stupid simple scheme where I split a file into 100 pieces and replicate each piece 3 times, I have a 100C1 * p^3 chance of irrecoverability where p is the independent probability of pieces dropping out. I guess I just want to know, given arbitrary p, what the probability of the file being gone is.
- GistNoesis 8y ago
  Error correction codes are quite magical. Instead of splitting in 100 pieces each piece 3 times, you split it in 300 distinct pieces of size 1/100 * filesize, which any 100 can reconstruct the file. So basically you can drop 2/3 of the pieces and yet be able to reconstruct the file. So as long as your nodes uptime is greater than 33% + eps, the file is not gone. The eps is quite small because of the law of great number. For example if your node uptime is 50%, the pieces will be in average available on 300 * 50% = 150 +- C*uptimevariance /sqrt(N) nodes which is greater than 100 with probability ~ 1, so the file will be recoverable almost surely.
  - eximius 8y ago
    Wow, Gokay codes were as advanced as I got but Fountain codes sound downright magical in comparison.

eximius 8y ago

I might need to look up how they work because that's basically what my next question is: how do the chunks overlap and how does that affect the probability of a file being unrecoverable? If I have a stupid simple scheme where I split a file into 100 pieces and replicate each piece 3 times, I have a 100C1 * p^3 chance of irrecoverability where p is the independent probability of pieces dropping out. I guess I just want to know, given arbitrary p, what the probability of the file being gone is.

GistNoesis 8y ago
Error correction codes are quite magical. Instead of splitting in 100 pieces each piece 3 times, you split it in 300 distinct pieces of size 1/100 * filesize, which any 100 can reconstruct the file. So basically you can drop 2/3 of the pieces and yet be able to reconstruct the file. So as long as your nodes uptime is greater than 33% + eps, the file is not gone. The eps is quite small because of the law of great number. For example if your node uptime is 50%, the pieces will be in average available on 300 * 50% = 150 +- C*uptimevariance /sqrt(N) nodes which is greater than 100 with probability ~ 1, so the file will be recoverable almost surely.
- eximius 8y ago
  Wow, Gokay codes were as advanced as I got but Fountain codes sound downright magical in comparison.