4 ms·
There are many similar gaps in collection in the early years of the Internet Archive. Early archive.org web captures came overwhelmingly from crawls by the com
by gojomo 7y ago
There are many similar gaps in collection in the early years of the Internet Archive.
Early archive.org web captures came overwhelmingly from crawls by the company Alexa Internet (which started in 1995 but by 2001 was a subsiiary of Amazon). Alexa's original bulk web crawling was intermittent: restarted once every couple of months, without much focus on rapid re-crawls of news sites. You can also see seasonal patterns, with larger lulls in summers.
The events of 2001-09-11 prompted a bunch of fresh, faster crawls – and higher prioritization of news website re-crawls.