4 ms·
I just tried the digg.com/robots.txt, and for it me it says: User-agent: * Disallow: Which explains everything without conspiracy theories
by perlgeek 14y ago
I just tried the digg.com/robots.txt, and for it me it says:
User-agent: *
Disallow:
Which explains everything without conspiracy theories
- philip1209 14y agoYes, this seems to explain it. http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html
- dredge 14y agoNo, it doesn't. Your linked example contains: User-agent: * Disallow: / While the (new) digg.com/robots.txt contains: User-agent: * Disallow: Those are very different. The former essentially disallows bots from crawling the entire site, while the latter disallows nothing - effectively allowing everything. The syntax is unusual, granted, for historical reasons.
- ecaron 14y agoA "Disallow:" (without anything after it) is saying allow all robots access to the entire server. (If you need something beyond my word on the topic, there's a thorough explanation at http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html)