4 ms·Googlebot also doesn't respect crawl delays in robots.txt.by smilliken 5y agoGooglebot also doesn't respect crawl delays in robots.txt.