4 ms·

> But layering robots-rules as a 'link' from the headers of the root page, or markup tags (if the root page happens to be HTML), still requires an initial inves

> But layering robots-rules as a 'link' from the headers of the root page, or markup tags (if the root page happens to be HTML), still requires an initial investigative hit to a site -- and if at the '/' page, probably involves more bandwidth than a compact robots.txt or 404. Not necessarily. If the location were specified in the http headers then a robot could use the HEAD command first. That's relatively minor bandwidth cost and still lets the robot get the site rules without first getting any content.

gojomo 17y ago
Using HEAD doesn't make a link-in-headers approach use less bandwidth than the classic /robots.txt convention. In the very best case -- no robots-rules -- a HEAD is equivalent in bandwidth used to a /robots.txt 404 without a content-body. But if there are robots-rules, the HEAD-then-robots approach means two hits and double-headers instead of one hit. (It's possible to strain and devise a convention that uses less bandwidth; that's why I said 'probably'. But that would require even more complexity -- such as the server being smart enough to send each robot only the rules that apply to it.) Configuring a server to emit special headers is also harder for most webmasters than dropping a text file into a conventional location. And a HEAD-then-robots process is more complicated for robots-writers. For the purpose of warning off robots, the /robots.txt placement convention was a very, very good solution on many axes, including minimizing traffic and adoption costs. Only a peculiar early-optimization based on a certain aesthetic sense -- and being concerned about being a bad example for other similar applications that aren't a pressing issue even now, 15 years later -- can justify Gregorio's opinion that /robots.txt "was a not-so-good idea when the robot exclusion protocol was rolled out".
- __david__ 17y ago
  Yeah, it's more bandwidth than the current "just ask for /robots.txt" method, but only epsilon more. Certainly orders of magnitude less than having the robots link in the html itself (since most index pages are multiple orders of magnitude more than the HTTP headers). The extra HEAD would be lost in the noise its bandwidth is so minimal. And you're right, the current scheme is easy. But it doesn't cover all the cases and makes certain types of sites impossible to do correctly (see http://news.ycombinator.com/item?id=639396 http://news.ycombinator.com/item?id=639396 for an example). Putting a link to the correct robots.txt for a URL in the HTTP headers would be way more flexible and only slightly raise the bandwidth cost. You could make it backwards compatible by assuming the current /robots.txt path if there is no header--So it would cost nothing to current site operators. Spider writers would have to do one extra HEAD but I refuse to believe that's terribly difficult.

gojomo 17y ago

Using HEAD doesn't make a link-in-headers approach use less bandwidth than the classic /robots.txt convention. In the very best case -- no robots-rules -- a HEAD is equivalent in bandwidth used to a /robots.txt 404 without a content-body. But if there are robots-rules, the HEAD-then-robots approach means two hits and double-headers instead of one hit. (It's possible to strain and devise a convention that uses less bandwidth; that's why I said 'probably'. But that would require even more complexity -- such as the server being smart enough to send each robot only the rules that apply to it.) Configuring a server to emit special headers is also harder for most webmasters than dropping a text file into a conventional location. And a HEAD-then-robots process is more complicated for robots-writers. For the purpose of warning off robots, the /robots.txt placement convention was a very, very good solution on many axes, including minimizing traffic and adoption costs. Only a peculiar early-optimization based on a certain aesthetic sense -- and being concerned about being a bad example for other similar applications that aren't a pressing issue even now, 15 years later -- can justify Gregorio's opinion that /robots.txt "was a not-so-good idea when the robot exclusion protocol was rolled out".

__david__ 17y ago
Yeah, it's more bandwidth than the current "just ask for /robots.txt" method, but only epsilon more. Certainly orders of magnitude less than having the robots link in the html itself (since most index pages are multiple orders of magnitude more than the HTTP headers). The extra HEAD would be lost in the noise its bandwidth is so minimal. And you're right, the current scheme is easy. But it doesn't cover all the cases and makes certain types of sites impossible to do correctly (see http://news.ycombinator.com/item?id=639396 http://news.ycombinator.com/item?id=639396 for an example). Putting a link to the correct robots.txt for a URL in the HTTP headers would be way more flexible and only slightly raise the bandwidth cost. You could make it backwards compatible by assuming the current /robots.txt path if there is no header--So it would cost nothing to current site operators. Spider writers would have to do one extra HEAD but I refuse to believe that's terribly difficult.