4 ms·

Very interesting experiment, and suprising that the Google bot appears (in this instance at least) to be ignoring robots.txt.

• zorked 15y ago
  I don't think it's ignoring robots.txt, it's probably just cached at another thread/machine/datacenter. It will pick up the new robots.txt eventually. This behavior is well-known.
  • afhof 15y ago
    But whats the point of it downloading the robots.txt file if it isn't going to honor it?
    • robtoo 15y ago
      Presumably, one cluster (or whatever google calls them) is honoring it, while a different cluster isn't.
• saalweachter 15y ago
  The problem is not Googlebot. robots.txt on ticker.picolisp.com says "Disallow: /", but ticker.picolisp.com redirects to picolisp.com/21000, and the robots.txt on picolisp.com says "Disallow:". If he wants Googlebot to stop crawling those URLs, he needs to add "Disallow: /21000" to picolisp.com.
  • yabannedaccount 15y ago
    No. GoogleBot needs to respect reality. We do not live in a world which serves Google. Google is supposed to serve us.
  • j_col 15y ago
    Hmmmm, maybe he just did: http://picolisp.com/robots.txt http://picolisp.com/robots.txt