Does it crawl URLs blocked by robots.txt? I doubt it. If you don't want well-behaving crawler to crawl your site, there's your answer. But not all are well beha

Does it crawl URLs blocked by robots.txt? I doubt it. If you don't want well-behaving crawler to crawl your site, there's your answer. But not all are well behaved...