Spammers' crawlers use URLs obtained from search engines and public sources. If the whole directory is blocked in robots.txt, it WILL reduce crawling activit

Spammers' crawlers use URLs obtained from search engines and public sources. If the whole directory is blocked in robots.txt, it WILL reduce crawling activity massively.