3 ms·
Robots.txt is respected retroactively?
by pygy_ 11y ago
Robots.txt is respected retroactively?
- gpvos 11y agoYes, but it is only temporary. As long as you have a robots.txt file excluding some URLs, those URLs will: 1) not be crawled by the Internet Archive crawler, 2) not be shown in the Wayback Machine. Any already-crawled pages will, however, invisibly remain in the archive, and will reappear once they are not in the robots.txt anymore.