4 ms·
Crawl-delay is not in the standard robots.txt protocol, and according to Wikipedia, some bots have different interpretations for this value. That's why maybe ma
by stummjr 10y ago
Crawl-delay is not in the standard robots.txt protocol, and according to Wikipedia, some bots have different interpretations for this value. That's why maybe many websites don't even bother defining the rate limits in robots.txt.
- greglindahl 10y agoI was referring to an actual rate limit, not crawl-delay. For example, YouTube is pretty strict about rate limits: http://www.bing.com/search?q=%22We+have+been+receiving+a+large+volume+of+requests+from+your+network%22+site%3Ayoutube.com http://www.bing.com/search?q=%22We+have+been+receiving+a+lar... I agree that crawl-delay is rare, and often it's set too long so that it's impossible to fully crawl a site -- as if the webmaster set it up 10 years ago and never updated it as their site got faster and bigger.