4 ms·
If you really care about durability, your best best is erasure-coding + a wide geographic distribution of shards. For example, you could encode 1 TB of data int
by nemo1618 11y ago
If you really care about durability, your best best is erasure-coding + a wide geographic distribution of shards. For example, you could encode 1 TB of data into four shards, each shard containing 500 GB. You distribute these to servers in SF, NYC, Berlin, and Sydney. The key here is that you only need two shards to recover your 1 TB of data, and they can be any two shards. So if lightning strikes Berlin, and the Big One hits SF, your data is still safe. And thanks to erasure-coding, you can achieve this with only 2x redundancy (instead of 4x).
- mirimir 11y agoIt's been a while since I followed this. I see over 100 Reed-Solomon erasure-coding projects on GitHub. Which would you recommend?
- nemo1618 11y agoWell, I have to shamelessly plug my startup, www.siacoin.com, which implements the scheme I described above using a peer-to-peer network, with payments made on a blockchain. We are using Klaus Post's excellent pure-Go implementation (https://github.com/klauspost/reedsolomon https://github.com/klauspost/reedsolomon) which exceeds 1GB/s throughput.
- mirimir 11y agoThank you. It's very cool. But can Sia work with hosts that are reachable only as Tor hidden services?