3 ms·
I am unfamiliar with the details of a robots.txt file. Is it possible to specify "I only want bots x, y and z to crawl my site"?
by scott_s 16y ago
I am unfamiliar with the details of a robots.txt file. Is it possible to specify "I only want bots x, y and z to crawl my site"?
- curio 16y agoyes, using the user-agent: http://en.wikipedia.org/wiki/Robots_exclusion_standard http://en.wikipedia.org/wiki/Robots_exclusion_standard
- scott_s 16y agoHow difficult is it for a bot to lie? Though if you've made it clear that only x, y and z can crawl your site, and someone spoofs, say, y, then it would be easy to demonstrate that someone has done something they know they shouldn't.
- curio 16y agoincredibly easy. and not only can the bot lie, it can disregard the robots.txt file altogether. just like the terms of service document for humans, you can choose to disregard it & deal w/ the consequences (blocked IP's, lawsuit, etc). robots.txt is just a version of the TOS that computers can read.