3 ms·
Not having things archived because you explicitly opted out of crawling is a feature, not a bug Otherwise you can whitelist a specific crawler in robots.txt
by procaryote 1y ago
Not having things archived because you explicitly opted out of crawling is a feature, not a bug
Otherwise you can whitelist a specific crawler in robots.txt
- rzzzt 1y ago(I understand it is a different entity) archive .org at one point started to honor the robots.txt settings of the website's current owner, hiding archived copies you could browse in the past. I don't know whether they still do this.