3 ms·

Maybe they nuked themselves: http://digg.com/robots.txt http://digg.com/robots.txt

by jcampbell1 14y ago

shanelja 14y ago
I noticed that too, they are also missing a sitemap.xml
kyrra 14y ago
getting 404 on that file.
fmavituna 14y ago
I'm pretty sure getting 404 from robots.txt will not get you nuked from Google :)
ErikAugust 14y ago
You do not need a robots.txt or a sitemap XML to be listed.
franze 14y ago
curl -I http://digg.com/robots.txt HTTP/1.1 404 Not Found Content-length: 1443 Content-Type: text/html Date: Wed, 20 Mar 2013 16:27:13 GMT Server: nginx/1.1.19 Connection: keep-alive an HTTP 404 robots.txt should not be an issue - but maybe there was a robots.txt with something else in there instead, and now it's gone. but even so, if there was an all blocking robots.txt site:digg.com would still show the URLs of digg.com, as crawling is optional for indexing. (maybe they used the non-documented Noindex: / directive, but i doubt it) if you want to achieve such a clean removal, in most cases you must request a complete removal via google webmaster tools. so yeah, somewhere, someone might have screwed up, most likely on diggs side, maybe on googles side, maybe a combination. UPDATe: now it's curl -i http://digg.com/robots.txt HTTP/1.1 200 OK Cache-Control: public Content-Type: text/plain Date: Wed, 20 Mar 2013 17:05:29 GMT Etag: "c47ccf1a49c24cc5842430aa75c72ef491292412" Last-Modified: Wed, 20 Mar 2013 16:51:48 GMT Server: TornadoServer/2.2 Content-Length: 24 Connection: keep-alive User-agent: * Disallow: which is the best practice robots.txt (allow all, or rather: hey, i have a valid robots.txt file but the second statement is not a valid robots.txt disallow directive, so this means allow all as there is no other disallow directive (note: i once coded https://npmjs.org/package/robotstxt https://npmjs.org/package/robotstxt which tried to reimplement googles spec of the robots.txt https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en https://developers.google.com/webmasters/control-crawl-index... so i have some experience in reading robots.txt files )) still, it wasn't a robots.txt issue hmmm, just a hypothesis: maybe just maybe someone thought it was a good idea to remove www.digg.com via google webmaster tools from google (as their main domain is digg.com and not www.digg.com and they definitely tried had some www.digg.com URL indexed, as even bing some some of these URLs indexed http://www.bing.com/search?q=site%3Awww.digg.com&go=&qs=ds&form=QBRE&filt=all http://www.bing.com/search?q=site%3Awww.digg.com&go=&#38... ) but google is (maybe) set to treat www.digg.com and digg.com as the same (via the settings panel in google webmaster tools), removing www.digg.com then could result into removing digg.com as well (something similar happened to a client of mine years ago), so could be the issue, could not be the issue, we would need more data (access to GWT) to verify this.
ryusage 14y ago
As another data point: I just clicked that link in Chrome, and I get the robots.txt with * Disallow like others are saying. Weird that some people are getting a 404.
highace 14y ago
It's cloaked so only google can see it - try changing your user agent to match googlebot's.
- cpeterso 14y ago
  Using the googlebot User-Agent string gives me: User-agent: * Disallow: This robots.txt should allow all bots to search the entire website. However, I think Google also penalizes websites that serve different content to googlebot than to non-bot user agents.
  - charonn0 14y ago
    This is the same as what I see, using a standard-issue Firefox UA string.
  - franze 14y ago
    they updated it
- blauwbilgorgel 14y ago
  That's weird because http://blog.digg.com/robots.txt http://blog.digg.com/robots.txt does not cloak. Why cloak one and leave the other? Sitemap: http://blog.digg.com/sitemap-pages.xml Sitemap: http://blog.digg.com/sitemap1.xml User-agent: * Disallow: /private Disallow: /random Disallow: /day Crawl-delay: 1
TobbenTM 14y ago
It's up for me with: User-agent: * Disallow:
RyanMcGreal 14y ago
It seems to be fixed now: User-agent: * Disallow:

Maybe they nuked themselves: http://digg.com/robots.txt http://digg.com/robots.txt

shanelja 14y ago

I noticed that too, they are also missing a sitemap.xml

kyrra 14y ago

getting 404 on that file.

fmavituna 14y ago

I'm pretty sure getting 404 from robots.txt will not get you nuked from Google :)

ErikAugust 14y ago

You do not need a robots.txt or a sitemap XML to be listed.

franze 14y ago

curl -I http://digg.com/robots.txt HTTP/1.1 404 Not Found Content-length: 1443 Content-Type: text/html Date: Wed, 20 Mar 2013 16:27:13 GMT Server: nginx/1.1.19 Connection: keep-alive an HTTP 404 robots.txt should not be an issue - but maybe there was a robots.txt with something else in there instead, and now it's gone. but even so, if there was an all blocking robots.txt site:digg.com would still show the URLs of digg.com, as crawling is optional for indexing. (maybe they used the non-documented Noindex: / directive, but i doubt it) if you want to achieve such a clean removal, in most cases you must request a complete removal via google webmaster tools. so yeah, somewhere, someone might have screwed up, most likely on diggs side, maybe on googles side, maybe a combination. UPDATe: now it's curl -i http://digg.com/robots.txt HTTP/1.1 200 OK Cache-Control: public Content-Type: text/plain Date: Wed, 20 Mar 2013 17:05:29 GMT Etag: "c47ccf1a49c24cc5842430aa75c72ef491292412" Last-Modified: Wed, 20 Mar 2013 16:51:48 GMT Server: TornadoServer/2.2 Content-Length: 24 Connection: keep-alive User-agent: * Disallow: which is the best practice robots.txt (allow all, or rather: hey, i have a valid robots.txt file but the second statement is not a valid robots.txt disallow directive, so this means allow all as there is no other disallow directive (note: i once coded https://npmjs.org/package/robotstxt https://npmjs.org/package/robotstxt which tried to reimplement googles spec of the robots.txt https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en https://developers.google.com/webmasters/control-crawl-index... so i have some experience in reading robots.txt files )) still, it wasn't a robots.txt issue hmmm, just a hypothesis: maybe just maybe someone thought it was a good idea to remove www.digg.com via google webmaster tools from google (as their main domain is digg.com and not www.digg.com and they definitely tried had some www.digg.com URL indexed, as even bing some some of these URLs indexed http://www.bing.com/search?q=site%3Awww.digg.com&go=&qs=ds&form=QBRE&filt=all http://www.bing.com/search?q=site%3Awww.digg.com&go=&#38... ) but google is (maybe) set to treat www.digg.com and digg.com as the same (via the settings panel in google webmaster tools), removing www.digg.com then could result into removing digg.com as well (something similar happened to a client of mine years ago), so could be the issue, could not be the issue, we would need more data (access to GWT) to verify this.

ryusage 14y ago

As another data point: I just clicked that link in Chrome, and I get the robots.txt with * Disallow like others are saying. Weird that some people are getting a 404.

highace 14y ago

It's cloaked so only google can see it - try changing your user agent to match googlebot's.

cpeterso 14y ago
Using the googlebot User-Agent string gives me: User-agent: * Disallow: This robots.txt should allow all bots to search the entire website. However, I think Google also penalizes websites that serve different content to googlebot than to non-bot user agents.
- charonn0 14y ago
  This is the same as what I see, using a standard-issue Firefox UA string.
- franze 14y ago
  they updated it
blauwbilgorgel 14y ago
That's weird because http://blog.digg.com/robots.txt http://blog.digg.com/robots.txt does not cloak. Why cloak one and leave the other? Sitemap: http://blog.digg.com/sitemap-pages.xml Sitemap: http://blog.digg.com/sitemap1.xml User-agent: * Disallow: /private Disallow: /random Disallow: /day Crawl-delay: 1

TobbenTM 14y ago

It's up for me with: User-agent: * Disallow:

RyanMcGreal 14y ago

It seems to be fixed now: User-agent: * Disallow: