2 ms·
A large fraction of websites with Crawl-Delay set it a decade ago and promptly forgot about it. No modern crawler uses it for anything other than a hint. The pr
by greglindahl 7y ago
A large fraction of websites with Crawl-Delay set it a decade ago and promptly forgot about it. No modern crawler uses it for anything other than a hint. The primary factors for crawl rate are usually site page count and response time.