3 ms·
Google does not support crawl delay. You can set your crawl rate lower every 90 days here https://www.google.com/webmasters/tools/settings https://www.google.c
by franze 4y ago
Google does not support crawl delay.
You can set your crawl rate lower every 90 days here https://www.google.com/webmasters/tools/settings https://www.google.com/webmasters/tools/settings
other possibilities are here https://support.google.com/webmasters/answer/48620?hl=en https://support.google.com/webmasters/answer/48620?hl=en
there is a way to lower crawling without total blocking (returning HTTP 503 if there is too much load)
- sorokod 4y agoI know it amounts to same, but from the users perspective it would be more accurate to say that Google doesn't respect crawl delay.
- franze 4y agocrawl delay was an extension of /robots.txt and never a part of the original "recommendation" and also never made it into the standard draft https://datatracker.ietf.org/doc/html/draft-rep-wg-topic-00#section-2.2 https://datatracker.ietf.org/doc/html/draft-rep-wg-topic-00#...