3 ms·

So its ok for them to do this? but if you do the same against google (scraping results) - presumably Bad Things happen?

• cstejerean 19y ago
  Is their search url listed in their robots.txt?
  • thorax 19y ago
    Seems strange that you wouldn't have just looked when asking that question (less characters to type at least). Lots of things disallowed. http://google.com/robots.txt http://google.com/robots.txt
    • Hexstream 19y ago
      I think it was a rhetorical question.
      • michaelneale 19y ago
        Yes it was. But I should have been clearer for the type of literal argument starters that would tend to hang around here ;) I wonder if google always honour robots.txt
        • Hexstream 19y ago
          I hope so because otherwise they could get stuck in one of those: http://en.wikipedia.org/wiki/Spider_trap http://en.wikipedia.org/wiki/Spider_trap