3 ms·

How about using Crawl-delay at robots.txt? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive http://en.wikipedia.org/wiki/Robots_exc

by o0Oo0O 13y ago

How about using Crawl-delay at robots.txt? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...