19 ms·
I just had a look and there's a non-standard "crawl-delay directive" extension to robots.txt that can be used to ask a spider to take some time between page vis
by YeGoblynQueenne 6y ago
I just had a look and there's a non-standard "crawl-delay directive" extension to robots.txt that can be used to ask a spider to take some time between page visits:
User-agent: bingbot
Allow : /
Crawl-delay: 10
https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...
- ricardo81 6y agoHello, MojeekBot doesn't observe the crawl-delay directive but thanks for the reminder of it as it's beneficial for us to know if site owners require more grace between requests.
- YeGoblynQueenne 6y agoHey. Good job with Mojeek. It seems the crawl-delay directive is not part of the robots.txt standard. It probably should be but that's not up to you!