2 ms·
However, the alternate argument is, how much control should the publisher have over the full manor in which that content is displayed to the end user? I guess,
by zaroth 12y ago
However, the alternate argument is, how much control should the publisher have over the full manor in which that content is displayed to the end user? I guess, this is what robots.txt is all about.
But where's the line to call something "robot"? A script run by a user, for their own use, I don't think it's bound by robots.txt. Does it matter who wrote the script?
- fenomas 12y agorobots.txt is nothing more than a way of politely asking the world not to recursively crawl through links on a site. It has no connection to whether someone is allowed to access the data or what they're allowed to do with it.