3 ms·
Right, now I remember - just double-checked, and Crawl-delay refers to seconds, not minutes. http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay
by konklone 13y ago
Right, now I remember - just double-checked, and Crawl-delay refers to seconds, not minutes.
http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...
- klapinat0r 13y agoOh, my bad. Sorry. Thanks for correcting me.