3 ms·
robots.txt is a nice way to make spiders ignore to index content, but doesn't imply not to access those
by _lce0 12y ago
robots.txt is a nice way to make spiders ignore to index content, but doesn't imply not to access those
- lern_too_spel 12y agoThat is not how robots.txt has been interpreted by any search engine. Robots.txt tells them not to access certain pages and says nothing about whether they should index those pages. (It's possible to index a page based solely on what links to it.) If you want your content crawled but not indexed, you should use the robots meta tag with a value of NOINDEX.