I was referring to an actual rate limit, not crawl-delay. For example, YouTube is pretty strict about rate limits: http://www.bing.com/search?q=%22We+have+been

I was referring to an actual rate limit, not crawl-delay. For example, YouTube is pretty strict about rate limits: http://www.bing.com/search?q=%22We+have+been+receiving+a+large+volume+of+requests+from+your+network%22+site%3Ayoutube.com http://www.bing.com/search?q=%22We+have+been+receiving+a+lar... I agree that crawl-delay is rare, and often it's set too long so that it's impossible to fully crawl a site -- as if the webmaster set it up 10 years ago and never updated it as their site got faster and bigger.