3 ms·
Doesn't Googlebot respect Crawl-Delay in robots.txt? PG has set it to be 30 secs - https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.t
by chintan 13y ago
Doesn't Googlebot respect Crawl-Delay in robots.txt? PG has set it to be 30 secs - https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt - which IMHO should not cause any load issues given the overall traffic profile of HN.
- kanamekun 13y agoGooglebot doesn't respect the crawl-delay setting in robots.txt. https://groups.google.com/forum/#!topic/Google_Webmaster_Help-Indexing/T5L9HUa3ZbQ https://groups.google.com/forum/#!topic/Google_Webmaster_Hel... As I understand it, the best way to lower the crawl rate is to log into Google Webmaster Tools and manually lower your crawl rate. The crawl rate delays expire every 90 days, so I set a calendar reminder to renew the crawl delay every 3 months. https://support.google.com/webmasters/answer/48620?hl=en https://support.google.com/webmasters/answer/48620?hl=en