4 ms·
Would be interesting to see stats on how often they had to resort to cold storage? Also, how do they go about thoroughly testing such a service? This is a main
by curiousDog 11y ago
Would be interesting to see stats on how often they had to resort to cold storage? Also, how do they go about thoroughly testing such a service? This is a main advantage of having 3 replicas, you can test HA in your statement in a straight-forward way. How does that work with the Reed-Solomon system? Also, kudos to Facebook for publishing articles to this to the public. A lot of companies keep this stuff secret.
- brettproctor 11y agoThis quote from the article talks about data verification. Although we all know that a backup isn't really a backup until you've tested restoring from it :-) """ To tackle this, we built a background “anti-entropy” process that detects data aberrations by periodically scanning all data on all the drives and reporting any detected corruptions. Given the inexpensive drives we would be using, we calculated that we should complete a full scan of all drives every 30 days or so to ensure we would be able to re-create any lost data successfully. Once an error was found and reported, another process would take over to read enough data to reconstruct the missing pieces and write them to new drives elsewhere. This separates the detection and root-cause analysis of the failure from reconstructing and protecting the data at hand. As a result of doing repairs in this distributed fashion, we were able to reduce reconstruction from hours to minutes. """