Yeah, it's more bandwidth than the current "just ask for /robots.txt" method, but only epsilon more. Certainly orders of magnitude less than having the robots l

Yeah, it's more bandwidth than the current "just ask for /robots.txt" method, but only epsilon more. Certainly orders of magnitude less than having the robots link in the html itself (since most index pages are multiple orders of magnitude more than the HTTP headers). The extra HEAD would be lost in the noise its bandwidth is so minimal. And you're right, the current scheme is easy. But it doesn't cover all the cases and makes certain types of sites impossible to do correctly (see http://news.ycombinator.com/item?id=639396 http://news.ycombinator.com/item?id=639396 for an example). Putting a link to the correct robots.txt for a URL in the HTTP headers would be way more flexible and only slightly raise the bandwidth cost. You could make it backwards compatible by assuming the current /robots.txt path if there is no header--So it would cost nothing to current site operators. Spider writers would have to do one extra HEAD but I refuse to believe that's terribly difficult.