4 ms·
Their robots.txt file clearly asks to not be crawled at all. User-agent: * Disallow: http://digg.com/robots.txt http://digg.com/robots.txt
by rodion_89 14y ago
Their robots.txt file clearly asks to not be crawled at all.
User-agent: *
Disallow:
http://digg.com/robots.txt http://digg.com/robots.txt
- blauwbilgorgel 14y agoIncorrect. To configure your robots.txt to not be crawled at all use: User-agent: * Disallow: / Allow indexing of everything with: User-agent: * Disallow: It seems they are at this very moment struggling to change things.
- rodion_89 14y agoYou are totally right. Disregard everything I said.
- chenster 14y agoDisallow nothing -> allow everything.