5 ms·
Backblaze uses Reed-Solomon erasure encoding. https://www.backblaze.com/blog/reed-solomon/ https://www.backblaze.com/blog/reed-solomon/ Loosely speaking files
by Twirrim 9y ago
Backblaze uses Reed-Solomon erasure encoding. https://www.backblaze.com/blog/reed-solomon/ https://www.backblaze.com/blog/reed-solomon/
Loosely speaking files are encoded in to to chunks and scattered across multiple disks (actual stored data is more than the size of the original object, but less than full duplication, triplication or other redundant ways to store it).
That blog suggests Backblaze use a Reed-Solomon ratio of 17:20 ratio, i.e. every object is encoded in to 20 chunks, for which you only need to retrieve 17 different chunks to be able to decode and get the full object.
If they're putting only one chunk per drive, that gives any object stored resilience from up to 3 disks failing, at a storage cost of only 1.17x original object size (20/17 ~= 1.176). They'll likely have repair processes to proactively handle any loss of shards from a disk failure, too.
Reed-Solomon is giving them something akin to triplication, while using significantly less disk space than duplicating.
Nothing comes for free, though. The additional cost here is the overhead of encoding and decoding the object, and on top of that the co-ordination etc required from requesting chunks from multiple locations. That adds some unavoidable latency, but that article suggest Backblaze find the JVM to be extremely fast (as fast as a pure C implementation of Reed-Solomon).
- 24gttghh 9y agoVery interesting! I was hoping to get some insight like this with my questions. Thank you!
- vxNsr 9y agoThis is more interesting than the actual post, I didn't even know about this type of encoding, thanks for giving me some weekend reading :)
- atYevP 9y agoYev from Backblaze here -> if you don't want to read it, we made this handy video -> https://www.youtube.com/watch?v=jgO09opx56o https://www.youtube.com/watch?v=jgO09opx56o