4 ms·

Thanks for this link. I've been digging for my old homepage data from '97 or so that started returning blanks on archive.org. I found a viable copy here!

walterbell 12y ago
If Alexandria was a backup of IA, shouldn't everything in Alexandria already be available at archive.org? Otherwise, that implies some pages were deleted or are no longer visible at archive.org.
- mmastrac 12y ago
  That's unfortunately the case. I've got pages that resolve as blank from archive.org and properly from Alexandria. I suspect there might be some data problems on the main archive.org server. They aren't deleted -- they show up in the timeline correctly -- but they come back as entirely blank when I load them (no toolbar, no nothing).
- gwern 12y ago
  IA hard drives fail, and as far as I know, they don't use anything fancy like FEC or distributed filesystems to heal from hard drives dying.
  - walterbell 12y ago
    Does the data eventually get restored from backup, when a hard drive fails?
  - lucb1e 12y ago
    They don't keep data twice? So the archive is just slowly rotting away literally, all the time? Okay, this just convinced me that Internetarchive.bak is a good idea. I'm going to participate.
    - gwern 12y ago
      I looked into this a few years ago when I noticed some of my IA links were erroring out, and yes, as far as I could tell, there is no redundancy in the IA's archives (except, obviously, at the snapshot level; if you lose a machine with some snapshots of cnn.com, there will probably be snapshots of cnn.com at other time periods on other machines). It's unfortunate, but can they really afford the extra space redundancy/FEC would necessarily require? Or the nontrivial software engineering to set up such redundancy? (Presumably their current servers are well-understood, well-debugged, and complex, while redundancy would bring in an unknowable number of new bugs.)
      - walterbell 12y ago
        Are there other online storage services (epecially one using the term 'archive') which loses data? This is hard to fathom. Why would version 1 of IA have been developed without a prime directive of not losing data? This is assumed by anyone working in the design of storage systems.
    - hbromley 12y ago
      Internet Archive engineer here. We do maintain two copies of everything in our main data collection. Disks fail at an average rate of perhaps 5-6 per day, and the content does get restored from the other copy. (That said, originally the Wayback crawl content was stored separately from paired storage, but it was merged into paired storage a few years ago.) We would prefer greater redundancy, and may shift to a more sophisticated arrangement at some point, but as a non-profit with limited resources, we're constantly managing tradeoffs between what we should do and what we can afford to do.
      - walterbell 12y ago
        Thanks for the reassuring clarification. Two copies is much better than one copy :) It would be useful to know the maximum time window for restoral from the backup copy.
        12y ago

walterbell 12y ago

If Alexandria was a backup of IA, shouldn't everything in Alexandria already be available at archive.org? Otherwise, that implies some pages were deleted or are no longer visible at archive.org.

mmastrac 12y ago
That's unfortunately the case. I've got pages that resolve as blank from archive.org and properly from Alexandria. I suspect there might be some data problems on the main archive.org server. They aren't deleted -- they show up in the timeline correctly -- but they come back as entirely blank when I load them (no toolbar, no nothing).
gwern 12y ago
IA hard drives fail, and as far as I know, they don't use anything fancy like FEC or distributed filesystems to heal from hard drives dying.
- walterbell 12y ago
  Does the data eventually get restored from backup, when a hard drive fails?
- lucb1e 12y ago
  They don't keep data twice? So the archive is just slowly rotting away literally, all the time? Okay, this just convinced me that Internetarchive.bak is a good idea. I'm going to participate.
  - gwern 12y ago
    I looked into this a few years ago when I noticed some of my IA links were erroring out, and yes, as far as I could tell, there is no redundancy in the IA's archives (except, obviously, at the snapshot level; if you lose a machine with some snapshots of cnn.com, there will probably be snapshots of cnn.com at other time periods on other machines). It's unfortunate, but can they really afford the extra space redundancy/FEC would necessarily require? Or the nontrivial software engineering to set up such redundancy? (Presumably their current servers are well-understood, well-debugged, and complex, while redundancy would bring in an unknowable number of new bugs.)
    - walterbell 12y ago
      Are there other online storage services (epecially one using the term 'archive') which loses data? This is hard to fathom. Why would version 1 of IA have been developed without a prime directive of not losing data? This is assumed by anyone working in the design of storage systems.
  - hbromley 12y ago
    Internet Archive engineer here. We do maintain two copies of everything in our main data collection. Disks fail at an average rate of perhaps 5-6 per day, and the content does get restored from the other copy. (That said, originally the Wayback crawl content was stored separately from paired storage, but it was merged into paired storage a few years ago.) We would prefer greater redundancy, and may shift to a more sophisticated arrangement at some point, but as a non-profit with limited resources, we're constantly managing tradeoffs between what we should do and what we can afford to do.
    - walterbell 12y ago
      Thanks for the reassuring clarification. Two copies is much better than one copy :) It would be useful to know the maximum time window for restoral from the backup copy.
      - 12y ago