3 ms·The Crawl-delay directive is the de facto standard for this.by timClicks 10y agoThe Crawl-delay directive is the de facto standard for this.