3 ms·
The problem is not Googlebot. robots.txt on ticker.picolisp.com says "Disallow: /", but ticker.picolisp.com redirects to picolisp.com/21000, and the robots.txt
by saalweachter 15y ago
The problem is not Googlebot.
robots.txt on ticker.picolisp.com says "Disallow: /", but ticker.picolisp.com redirects to picolisp.com/21000, and the robots.txt on picolisp.com says "Disallow:". If he wants Googlebot to stop crawling those URLs, he needs to add "Disallow: /21000" to picolisp.com.
- yabannedaccount 15y agoNo. GoogleBot needs to respect reality. We do not live in a world which serves Google. Google is supposed to serve us.
- j_col 15y agoHmmmm, maybe he just did: http://picolisp.com/robots.txt http://picolisp.com/robots.txt