2 ms·
This is a surprisingly primitive and simple system. A twice replicated file is a huge space overhead for not very reliable storage. (I mean the probability of l
by kevincox 10y ago
This is a surprisingly primitive and simple system. A twice replicated file is a huge space overhead for not very reliable storage. (I mean the probability of losing two disks at the same time isn't high, but it's also not immeasurably low)
For example fb talked about how they use erasure coding to use 1.4x space overhead for far higher reliability.
https://code.facebook.com/posts/1433093613662262/-under-the-hood-facebook-s-cold-storage-system-/ https://code.facebook.com/posts/1433093613662262/-under-the-...
While erasure coding can have performance impact (although mostly only on failures) a multi-tier storage system will eliminate that. I presume that most attachments aren't read more the a week or two after being received.
So while it is definitely nice to get some savings through deduplication there is a lot more gains that can be had by more intelligent storage of the data.