3 ms·
Google specifically recommended using robots.txt in Copiepresse v. Google. I haven't noticed them disobeying robots.txt on my own sites.
by lern_too_spel 12y ago
Google specifically recommended using robots.txt in Copiepresse v. Google. I haven't noticed them disobeying robots.txt on my own sites.
- smilliken 12y agoI've seen several blog posts on exceptional cases before, but the only case I've personally witnessed is how the crawl-delay directive is ignored. Granted you can set it if you're a webmaster tools user, but even then the setting disables itself after a few months.