2 ms·
Their robots.txt shows they allowed Google (at an extreme 3-minutes-between-fetches crawl-delay), but no other search engines, nor the Internet Archive. Otherw
by gojomo 18y ago
Their robots.txt shows they allowed Google (at an extreme 3-minutes-between-fetches crawl-delay), but no other search engines, nor the Internet Archive.
Otherwise, a tool like 'Warrick' might get back a significant portion of the content -- especially any that was old and well-inlinked. As it stands, maybe they can scrape a little from Google's cache.
Warrick: http://warrick.cs.odu.edu/ http://warrick.cs.odu.edu/