3 ms·

Is that part of the robots.txt RFC or just a happy security coincidence? The bit.ly issue seems like a pretty bad bug that ought to be fixed. You also might be

by mustpax 17y ago

Is that part of the robots.txt RFC or just a happy security coincidence? The bit.ly issue seems like a pretty bad bug that ought to be fixed. You also might be able to fool Google Webmaster Tools or other utilities into thinking you own bit.ly. One of the authentication methods I've seen used was the creation of an empty html document in the root directory.

gojomo 17y ago
The 1994 original proposal is silent on what to do with any unexpected response codes/types. The 1997 internet-draft (never an RFC) suggests the redirect should be followed, but the potential confusion that could cause has meant few, if any, crawlers have followed that guidance. It's more likely to be a webmaster configuration error than real intent.