4 ms·
Internet Archive engineer here. We do maintain two copies of everything in our main data collection. Disks fail at an average rate of perhaps 5-6 per day, and t
by hbromley 12y ago
Internet Archive engineer here. We do maintain two copies of everything in our main data collection. Disks fail at an average rate of perhaps 5-6 per day, and the content does get restored from the other copy.
(That said, originally the Wayback crawl content was stored separately from paired storage, but it was merged into paired storage a few years ago.)
We would prefer greater redundancy, and may shift to a more sophisticated arrangement at some point, but as a non-profit with limited resources, we're constantly managing tradeoffs between what we should do and what we can afford to do.
- walterbell 12y agoThanks for the reassuring clarification. Two copies is much better than one copy :) It would be useful to know the maximum time window for restoral from the backup copy.
- hbromley 12y agoDepending on what time of day it fails, the bad disk is generally replaced within 2-18 hours. Copying the data onto the new disk, from the remaining half of the original pair, typically takes another day or so. But the material found on the failed disk usually remains available throughout the replacement process because both original copies are on live servers. Read requests are ordinarily divided more or less evenly between the two. If one copy dies, requests are simply all directed to the other copy until the lost one is replaced. There is no ability to "write" (make changes) to any affected items during the replacement process, but they usually remain readable throughout.