4 ms·
Erasure encoding does not greatly increase storage space consumption. Consider a 2n or 3n replication factor in systems like hdfs, Cassandra and Riak. In those
by siculars 10y ago
Erasure encoding does not greatly increase storage space consumption. Consider a 2n or 3n replication factor in systems like hdfs, Cassandra and Riak. In those cases storage space consumption is 200% and 300%, respectively. Erasure encoding allows you to select the percentage increase in space consumed for some level of durability. You can, for instance, confirgure the system to encode a given blob of data at 1.5x (150%) space consumption and then stripe that data across 10 or so locations, only requiring 6 or 7 of them to reconstruct the original blob. Facebook put a post out about this some while back re. their long term picture storage.
- lomnakkus 10y agoErase encoding was my first thought too. Here's a link to the paper, for anyone who's interested: https://research.fb.com/publications/xoring-elephants-novel-erasure-codes-for-big-data/ https://research.fb.com/publications/xoring-elephants-novel-...
- adt2bt 10y agoYou're correct, I mistyped & intended to say what you wrote.