3 ms·

It's cloaked so only google can see it - try changing your user agent to match googlebot's.

• cpeterso 14y ago
  Using the googlebot User-Agent string gives me: User-agent: * Disallow: This robots.txt should allow all bots to search the entire website. However, I think Google also penalizes websites that serve different content to googlebot than to non-bot user agents.
  • charonn0 14y ago
    This is the same as what I see, using a standard-issue Firefox UA string.
  • franze 14y ago
    they updated it
• blauwbilgorgel 14y ago
  That's weird because http://blog.digg.com/robots.txt http://blog.digg.com/robots.txt does not cloak. Why cloak one and leave the other? Sitemap: http://blog.digg.com/sitemap-pages.xml Sitemap: http://blog.digg.com/sitemap1.xml User-agent: * Disallow: /private Disallow: /random Disallow: /day Crawl-delay: 1