3 ms·
Oh I am absolutely aware that erasure codes are an old thing, Reed-Solomon codes exist since the 1960's, but this is not simply a trivial application of erasure
by luizfelberti 6y ago
Oh I am absolutely aware that erasure codes are an old thing, Reed-Solomon codes exist since the 1960's, but this is not simply a trivial application of erasure coding to a storage system: erasure codes alone don't provide the same durability guarantees that replication does. [0]
This is a combination of erasure coding AND replication, whose combined storage amplification is dramatically lower than previous SOTA.
I gave a longer explanation in a sibing comment to yours [1]
[0] http://fireless.cs.cornell.edu/publications/erasure_iptps.pdf http://fireless.cs.cornell.edu/publications/erasure_iptps.pd...
[1] https://news.ycombinator.com/item?id=25351678 https://news.ycombinator.com/item?id=25351678
- jeffbee 6y agoThanks for the clarification. I still think these techniques were somewhat widespread already ... see for example this figure from US Patent 9292389 describing nested, aka layered coding that to my thinking is isomorphic with the "LRC" or "pyramid code" described by the Microsoft researchers. By the way, not at all trying to say this paper isn't interesting. I keep it in my filing cabinet to show my colleagues when I need to describe this technique, since Google hasn't ever bothered to describe Colossus in a way I can reference. https://imgur.com/a/gi2Xdl0 https://imgur.com/a/gi2Xdl0