3 ms·
Wait so adding a robots.txt exclusion for the Wayback Machine makes all previous archives of the site inaccessible? That's very odd behaviour, and really not th
by parliament32 9y ago
Wait so adding a robots.txt exclusion for the Wayback Machine makes all previous archives of the site inaccessible? That's very odd behaviour, and really not the point of a robots.txt file... I would expect a robots.txt to control a bot's visits / scraping behaviour, not a site's history.
- adventured 9y agoYes. They use the robots.txt file to essentially ascertain ownership or control of the domain. Archive.org doesn't want to delete the content they have (for whatever reason), so the compromise they came up with is to read the robots.txt and then hide the content they have archived if the present domain owner/controller wants it to be that way. If you remove the robots.txt setting, the archives become available again.