4 ms·
A bit surprised at the comments here. As far as I can remember, robots.txt has always been used by the Wayback Machine this way. Often a bummer when the origina
by BeetleB 9y ago
A bit surprised at the comments here. As far as I can remember, robots.txt has always been used by the Wayback Machine this way. Often a bummer when the original domain expires and a domain squatter takes over - many have robots.txt
- dredmorbius 9y agoThat's changed. A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly. We see the future of web archiving relying less on robots.txt file declarations geared toward search engines, and more on representing the web as it really was, and is, from a user’s perspective. https://teleread.org/2017/04/24/the-internet-archive-will-soon-stop-honoring-robots-txt-files/ https://teleread.org/2017/04/24/the-internet-archive-will-so... https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...