3 ms·
Except for when someone gives them a robots.txt instruction. Then they forget whole websites.
by J_Darnley 10y ago
Except for when someone gives them a robots.txt instruction. Then they forget whole websites.
- CM30 10y agoYeah, which is the Internet Archive's biggest issue. What site uses a domain name at one time isn't what site might use the same domain name at a different date. If the archive had more common sense, they'd note down when sites and robots.txt files change and only block pages by the site owner with the offending robots.txt file. So if say, Google went under and a new company started up, a robots.txt file from the latter would only block files added since the site switched over. That would solve 99.99% of these issues.
- db48x 10y agoThe IA doesn't forget whole websites. If the current robots.txt file excludes some website from crawling, IA hides (but doesn't delete), the excluded information.