3 ms·TIL robots.txt has a crawl-delay directive. Thank you.by mooism2 13y agoTIL robots.txt has a crawl-delay directive. Thank you.