3 ms·
it seems weird to me to retroactively obey robots.txt for a _different_ version of the site. It makes much more sense to archive the robots.txt along with the
by chii 10y ago
it seems weird to me to retroactively obey robots.txt for a _different_ version of the site.
It makes much more sense to archive the robots.txt along with the content, and use the robots.txt linked with that version of the site. Updating the current robots.txt shouldn't affect a past archive.
- foldor 10y agoI believe the reasoning is that if I accidentally published a public page and didn't realize it but later fixed the problem, I'd like to make sure no one else could see it later. It's probably easier than dealing with all kinds of takedown requests.