3 ms·
Googlebot doesn't use the crawl delay line in robots.txt if that's what you're asking. I guess theoretically it's because you can normally change it in the web
by corobo 12y ago
Googlebot doesn't use the crawl delay line in robots.txt if that's what you're asking. I guess theoretically it's because you can normally change it in the webmasters interface.
It's not really ignoring robots.txt either, as crawl delay is not an 'official' setting