2 ms·

Yes, this seems to explain it. http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html

• dredge 14y ago
  No, it doesn't. Your linked example contains: User-agent: * Disallow: / While the (new) digg.com/robots.txt contains: User-agent: * Disallow: Those are very different. The former essentially disallows bots from crawling the entire site, while the latter disallows nothing - effectively allowing everything. The syntax is unusual, granted, for historical reasons.