5 ms·
A minimum of 4 seconds.
by ricardo81 6y ago
A minimum of 4 seconds.
- YeGoblynQueenne 6y agoI just had a look and there's a non-standard "crawl-delay directive" extension to robots.txt that can be used to ask a spider to take some time between page visits: User-agent: bingbot Allow : / Crawl-delay: 10 https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...
- ricardo81 6y agoHello, MojeekBot doesn't observe the crawl-delay directive but thanks for the reminder of it as it's beneficial for us to know if site owners require more grace between requests.
- YeGoblynQueenne 6y agoHey. Good job with Mojeek. It seems the crawl-delay directive is not part of the robots.txt standard. It probably should be but that's not up to you!