3 ms·
As I recall, this is outdated information. Internet Archive does respect robots.txt and will remove a site from its archive based on robots.txt. I have done thi
by AnonC 2y ago
As I recall, this is outdated information. Internet Archive does respect robots.txt and will remove a site from its archive based on robots.txt. I have done this a few years after your linked blog post to get an inconsequential site removed from archive.org.
- dredmorbius 2y agoThe most recent notice IA have blogged was in 2017, and there's no indication that the service has reversed course on robots.txt since. <https://blog.archive.org/?s=robots.txt https://blog.archive.org/?s=robots.txt>