3 ms·Do they not respect the crawl-delay directive in robots.txt?by phreeza 2y agoDo they not respect the crawl-delay directive in robots.txt?