4 ms·
that's what robots.txt is for and works well Robots.txt doesn't stop anyone doing anything, it is simply a policy. Bots either respect it or they do not. Arc
by chalst 15y ago
that's what robots.txt is for and works well
Robots.txt doesn't stop anyone doing anything, it is simply a policy. Bots either respect it or they do not. Archiveteam.org have indicated that they wish to join the side of the spammers and incompetent spider authors.
- user24 15y agoare you confusing archive.org and archiveteam?
- chalst 15y agoThanks, fixed. Actually I was not, that was a typo.
- joelmichael 15y agoI agree. archiveteam.org have stated they will ignore robots.txt. This makes me disrespect their organization. Well, that and their ridiculous website.