3 ms·
robots.txt is simple and effective. I do not want certain bots, especially so-called "archives" to automatically download all my content. And that's what robot
by yaix 15y ago
robots.txt is simple and effective.
I do not want certain bots, especially so-called "archives" to automatically download all my content. And that's what robots.txt is for and works well.
The article is just stupid, sorry. There is not one real knowledgable argument.
- chalst 15y agothat's what robots.txt is for and works well Robots.txt doesn't stop anyone doing anything, it is simply a policy. Bots either respect it or they do not. Archiveteam.org have indicated that they wish to join the side of the spammers and incompetent spider authors.
- user24 15y agoare you confusing archive.org and archiveteam?
- chalst 15y agoThanks, fixed. Actually I was not, that was a typo.
- joelmichael 15y agoI agree. archiveteam.org have stated they will ignore robots.txt. This makes me disrespect their organization. Well, that and their ridiculous website.