3 ms·
Yes, this. The whole purpose of robots.txt is to allow restrictions to be applied on objects that the crawler hasn't looked at yet. A META tag can only apply re
by duskwuff 17y ago
Yes, this. The whole purpose of robots.txt is to allow restrictions to be applied on objects that the crawler hasn't looked at yet. A META tag can only apply restrictions after the fact.
Moreover, his proposed replacement is incredibly browser-centric. It requires any compliant crawler to contain an HTML parser. And woe befall anyone who typos the META tag, or manages to confuse the HTML parser before it gets there! The robots.txt specification, by contrast, requires no such heavy lifting: just request a single fixed URL.
META tags make perfect sense for favicon and the like - I won't dispute that. Robots exclusion, however, is a special case - it belongs outside HTML, not inside it.