3 ms·

I think the biggest question here is, if a site robots.txt file prohibits bots from gathering data, should that also prohibit bots that piggyback on real user s

• pbhjpbhj 16y ago
  >if a site robots.txt file prohibits bots from gathering data, Pages in a robots.txt file still get put into Google's index they just don't get parsed. Surprised me to find that out; http://www.seomoz.org/learn-seo/robotstxt http://www.seomoz.org/learn-seo/robotstxt (see "Why Meta Robots is better than robots.txt").
• user24 16y ago
  Good question. robots.txt was designed for an age before ajax and the proliferation of plugins/toolbars/etc. It's a valid shortcoming you've identified, imho.
• RobGR 16y ago
  I think it's an interesting question, but I think it will be technically difficult to differentiate between a person using data (say, with a plugin that indexes everything they browse, or a separate program that indexes their browser cache) and something like the Bing toolbar.
• lazugod 16y ago
  Agreed. That ambiguity is why Google's PR response makes sense.