4 ms·
In practice robots.txt is just a suggestion; even Google doesn't fully respect it.
by smilliken 12y ago
In practice robots.txt is just a suggestion; even Google doesn't fully respect it.
- lern_too_spel 12y agoGoogle specifically recommended using robots.txt in Copiepresse v. Google. I haven't noticed them disobeying robots.txt on my own sites.
- smilliken 12y agoI've seen several blog posts on exceptional cases before, but the only case I've personally witnessed is how the crawl-delay directive is ignored. Granted you can set it if you're a webmaster tools user, but even then the setting disables itself after a few months.
- _lce0 12y agorobots.txt is a nice way to make spiders ignore to index content, but doesn't imply not to access those
- lern_too_spel 12y agoThat is not how robots.txt has been interpreted by any search engine. Robots.txt tells them not to access certain pages and says nothing about whether they should index those pages. (It's possible to index a page based solely on what links to it.) If you want your content crawled but not indexed, you should use the robots meta tag with a value of NOINDEX.