4 ms·

robots.txt isn't enforced.

• easy_rider 13y ago
  Maybe they should be. Gentleman's agreements do not apply to robots.
  • cheald 13y ago
    And how exactly do you propose verifying that the user agent purporting to be Googlebot or Firefox is actually who they are? They're inherently unenforceable. robots.txt is basically a list of rules that lay out "This is how we'd like you to crawl us. We might stop serving you if you don't comply", rather than a hard-and-fast set of directives that specify how a webcrawler will be guaranteed to behave.
    • easy_rider 13y ago
      You can implement some strict enforcing in Apache using some crafty mod_rewrite stuff: http://andthatsjazz.org/defeat.html http://andthatsjazz.org/defeat.html User-agent is to easily spoofed, but we could check if the robots are indeed Google (whitelisted) and not some other crawler that just wants to scrape your content. In the realm of mail servers we have something called SPF: http://en.wikipedia.org/wiki/Sender_Policy_Framework http://en.wikipedia.org/wiki/Sender_Policy_Framework Just thinking out of the box here, but other than checking IP ranges: Maybe a hash being sent as a header inside the GET request by the crawler to verify if they are who they say they are.