4 ms·
IA hard drives fail, and as far as I know, they don't use anything fancy like FEC or distributed filesystems to heal from hard drives dying.
by gwern 12y ago
IA hard drives fail, and as far as I know, they don't use anything fancy like FEC or distributed filesystems to heal from hard drives dying.
- walterbell 12y agoDoes the data eventually get restored from backup, when a hard drive fails?
- lucb1e 12y agoThey don't keep data twice? So the archive is just slowly rotting away literally, all the time? Okay, this just convinced me that Internetarchive.bak is a good idea. I'm going to participate.
- gwern 12y agoI looked into this a few years ago when I noticed some of my IA links were erroring out, and yes, as far as I could tell, there is no redundancy in the IA's archives (except, obviously, at the snapshot level; if you lose a machine with some snapshots of cnn.com, there will probably be snapshots of cnn.com at other time periods on other machines). It's unfortunate, but can they really afford the extra space redundancy/FEC would necessarily require? Or the nontrivial software engineering to set up such redundancy? (Presumably their current servers are well-understood, well-debugged, and complex, while redundancy would bring in an unknowable number of new bugs.)
- walterbell 12y agoAre there other online storage services (epecially one using the term 'archive') which loses data? This is hard to fathom. Why would version 1 of IA have been developed without a prime directive of not losing data? This is assumed by anyone working in the design of storage systems.
- hbromley 12y agoInternet Archive engineer here. We do maintain two copies of everything in our main data collection. Disks fail at an average rate of perhaps 5-6 per day, and the content does get restored from the other copy. (That said, originally the Wayback crawl content was stored separately from paired storage, but it was merged into paired storage a few years ago.) We would prefer greater redundancy, and may shift to a more sophisticated arrangement at some point, but as a non-profit with limited resources, we're constantly managing tradeoffs between what we should do and what we can afford to do.
- walterbell 12y agoThanks for the reassuring clarification. Two copies is much better than one copy :) It would be useful to know the maximum time window for restoral from the backup copy.
- hbromley 12y agoDepending on what time of day it fails, the bad disk is generally replaced within 2-18 hours. Copying the data onto the new disk, from the remaining half of the original pair, typically takes another day or so. But the material found on the failed disk usually remains available throughout the replacement process because both original copies are on live servers. Read requests are ordinarily divided more or less evenly between the two. If one copy dies, requests are simply all directed to the other copy until the lost one is replaced. There is no ability to "write" (make changes) to any affected items during the replacement process, but they usually remain readable throughout.