4 ms·

That one got even picked up by wired, this also contains an interview with one of their developers at the time http://www.wired.com/epicenter/2010/08/robot-laws

• someone13 15y ago
  Good catch - didn't even know that. I came across this thanks to a tweet by Mikko Hypponen[1]. It was one of the top three or four. Fox News's robots.txt[2] is pretty interesting too - they deliberately block at least one article from being indexed[3] And YouTube[4], as usual, shows some humor in the form of Flight Of The Conchords lyrics. [1] http://twitter.com/#!/mikkohypponen/status/103440803132747776 http://twitter.com/#!/mikkohypponen/status/10344080313274777... [2] http://www.foxnews.com/robots.txt http://www.foxnews.com/robots.txt [3] http://www.foxnews.com//politics/2011/07/21/wynn-slams-obama-for-scaring-businesses-as-company-profits-soar/ http://www.foxnews.com//politics/2011/07/21/wynn-slams-obama... [4] http://www.youtube.com/robots.txt http://www.youtube.com/robots.txt