4 ms·
The largest problem with the only-use-the-meta-tags solution is that the page to be excluded has to be retrieved before it can be excluded.
by michaelfairley 17y ago
The largest problem with the only-use-the-meta-tags solution is that the page to be excluded has to be retrieved before it can be excluded.
- motherwell 17y agoJust for clarrification: 1. robots.txt excludes CRAWLING e.g. downloading but NOT indexing, e.g. including a URL / site in a database of known URLs / sites. 2. Robots meta tag disallows INDEXING but NOT crawling. So it is semantically correct, although most modern SEs do not do this, to index a site / URL that is disallowed via robots.txt, using link data alone.
- duskwuff 17y agoYes, this. The whole purpose of robots.txt is to allow restrictions to be applied on objects that the crawler hasn't looked at yet. A META tag can only apply restrictions after the fact. Moreover, his proposed replacement is incredibly browser-centric. It requires any compliant crawler to contain an HTML parser. And woe befall anyone who typos the META tag, or manages to confuse the HTML parser before it gets there! The robots.txt specification, by contrast, requires no such heavy lifting: just request a single fixed URL. META tags make perfect sense for favicon and the like - I won't dispute that. Robots exclusion, however, is a special case - it belongs outside HTML, not inside it.