Robots.txt has a directive for crawl delay, and like all directives it can be applied to specific user agents. In addition to that, the webmaster console can b

Robots.txt has a directive for crawl delay, and like all directives it can be applied to specific user agents. In addition to that, the webmaster console can be used to set the crawl speed, and that data would obviously not be readable to third parties.