3 ms·
I'm surprised Erasure Coding didn't come up here. It greatly increases durability and storage space required at the cost of higher latency reads. https://en.wi
by adt2bt 10y ago
I'm surprised Erasure Coding didn't come up here. It greatly increases durability and storage space required at the cost of higher latency reads.
https://en.wikipedia.org/wiki/Erasure_code https://en.wikipedia.org/wiki/Erasure_code
- siculars 10y agoErasure encoding does not greatly increase storage space consumption. Consider a 2n or 3n replication factor in systems like hdfs, Cassandra and Riak. In those cases storage space consumption is 200% and 300%, respectively. Erasure encoding allows you to select the percentage increase in space consumed for some level of durability. You can, for instance, confirgure the system to encode a given blob of data at 1.5x (150%) space consumption and then stripe that data across 10 or so locations, only requiring 6 or 7 of them to reconstruct the original blob. Facebook put a post out about this some while back re. their long term picture storage.
- lomnakkus 10y agoErase encoding was my first thought too. Here's a link to the paper, for anyone who's interested: https://research.fb.com/publications/xoring-elephants-novel-erasure-codes-for-big-data/ https://research.fb.com/publications/xoring-elephants-novel-...
- adt2bt 10y agoYou're correct, I mistyped & intended to say what you wrote.