2 ms·
That is not how robots.txt has been interpreted by any search engine. Robots.txt tells them not to access certain pages and says nothing about whether they shou
by lern_too_spel 12y ago
That is not how robots.txt has been interpreted by any search engine. Robots.txt tells them not to access certain pages and says nothing about whether they should index those pages. (It's possible to index a page based solely on what links to it.) If you want your content crawled but not indexed, you should use the robots meta tag with a value of NOINDEX.