3 ms·
This is true, however they don't purge any of the old archives. If the robots.txt is updated to allow bots again then all of the previous archives will become a
by davegardner 10y ago
This is true, however they don't purge any of the old archives. If the robots.txt is updated to allow bots again then all of the previous archives will become available again.
- chii 10y agoit seems weird to me to retroactively obey robots.txt for a _different_ version of the site. It makes much more sense to archive the robots.txt along with the content, and use the robots.txt linked with that version of the site. Updating the current robots.txt shouldn't affect a past archive.
- foldor 10y agoI believe the reasoning is that if I accidentally published a public page and didn't realize it but later fixed the problem, I'd like to make sure no one else could see it later. It's probably easier than dealing with all kinds of takedown requests.