2 ms·
It's telling googlebot to slow down and not be so aggressive. You can use google webmaster tools to dial down the crawl rate. Some think you can use Crawl-del
by newman314 15y ago
It's telling googlebot to slow down and not be so aggressive.
You can use google webmaster tools to dial down the crawl rate.
Some think you can use Crawl-delay: in robots.txt but I have not tested this (and it may very well not work).