2 ms·

The solution to this is... robots.txt. Otherwise your site might turn up in Google etc. Since it's archive.org that's doing the crawling and they respect robots

The solution to this is... robots.txt. Otherwise your site might turn up in Google etc. Since it's archive.org that's doing the crawling and they respect robots.txt it won't get archived.

symfoniq 6y ago
Archive.org does not respect robots.txt IIRC. I’ve run into this problem before with them. Ironically, I ended up blocking Internet Archive’s ASN using Cloudflare. EDIT: Internet Archive started ignoring robots.txt in 2017: https://www.digitaltrends.com/computing/internet-archive-robots-txt/ https://www.digitaltrends.com/computing/internet-archive-rob...
- kalleboo 6y ago
  They only started ignoring robots.txt on US government websites (as that article also says)
  - symfoniq 6y ago
    That is not what the article says. It says Internet Archive had already started ignoring robots.txt on US government websites. Now (since 2017) they ignore it on all websites.