3 ms·
I know it amounts to same, but from the users perspective it would be more accurate to say that Google doesn't respect crawl delay.
by sorokod 4y ago
I know it amounts to same, but from the users perspective it would be more accurate to say that Google doesn't respect crawl delay.
- franze 4y agocrawl delay was an extension of /robots.txt and never a part of the original "recommendation" and also never made it into the standard draft https://datatracker.ietf.org/doc/html/draft-rep-wg-topic-00#section-2.2 https://datatracker.ietf.org/doc/html/draft-rep-wg-topic-00#...