3 ms·
And then finding that the page is blocked by a robots.txt that is present now, but probably wasn't when the page originally existed.
by gpvos 11y ago
And then finding that the page is blocked by a robots.txt that is present now, but probably wasn't when the page originally existed.
- pygy_ 11y agoRobots.txt is respected retroactively?
- gpvos 11y agoYes, but it is only temporary. As long as you have a robots.txt file excluding some URLs, those URLs will: 1) not be crawled by the Internet Archive crawler, 2) not be shown in the Wayback Machine. Any already-crawled pages will, however, invisibly remain in the archive, and will reappear once they are not in the robots.txt anymore.