3 ms·
Silly question, but wouldn't putting something like Crawl-delay in the robots.txt file (somewhat) help alleviate the situation (if it is respected)? Or maybe e
by lancer383 12y ago
Silly question, but wouldn't putting something like Crawl-delay in the robots.txt file (somewhat) help alleviate the situation (if it is respected)?
Or maybe even just block crawling of the entire site except for the homepage?
- thekeywordgeek 12y agoGoogle doesn't respect crawl-delay, sadly. They rely on the Webmaster Tools setting, which is unavailable to me as I've described.