4 ms·
Once it's stored I imagine they don't need to even scrape the page again, so robots.txt wouldn't do anything.
by Exuma 9y ago
Once it's stored I imagine they don't need to even scrape the page again, so robots.txt wouldn't do anything.
- deleted 9y ago[deleted]
- dragonwriter 9y agoInternet archive does rescrape periodically, and it removes archived pages based on the current robots.txt. This behavior is documented behavior of the archive that goes beyond the normal conventions of robots.txt.
- LinuxBender 8y agoI would add, the content itself is not removed. They only stop displaying it whilst the robots.txt says not to. If they can not reach your robots.txt, the content comes back as I have experienced multiple times.