3 ms·
The article is down, but here is the text from Google's cache: --snip-- Something interesting has just come across one of my networks (hat tip to datadia
by MiguelHudnandez 14y ago
The article is down, but here is the text from Google's cache:
--snip--
Something interesting has just come across one of my networks (hat tip to datadial), just a few days after Digg have announced that they are building a replacement for the much loved Google Reader, they have (coincidentally?) disappeared from the primary google index.
[image unavailable]
Is it an SEO penalty for links? That seems to be the number one reason that brands are getting booted from google’s index these days… Some conspiracy theorists will no doubt be proclaiming its something to do with their announcement to build a replica of the now defunct Google reader, but personally I really cant see that having any effect. Could there?
Doing a site: search for Digg certainly demonstrates that they are no longer in the index:
[image unavailable]
Its likely (only if it is link based however) that it would be down to what individuals who submit content do after the fact – ie. sending spammy links at their posts to try and build the pagerank, and create “authority” which they then pass back to their own sites. Digg has long been listed in every “linkwheel” sellers handbook, and if that is the reason then what does it mean for every community site on the internet?
Will we have to manually aprove all new links soon at this rate? Come on Google – WTF – let the internet know what you’re doing please.
--end snip--
Without the images, maybe I am missing some context. But it seems hyperbolic, considering digg is not serving a robots.txt anymore [1]. It is probably just a blunder on Digg's part.
[1]: At the time of my comment, digg was serving an xhtml document with status 404 at /robots.txt. Now it appears to be a valid robots file.
PS: I am enjoying the irony of having a copy of the article, even though the site is down, because of Google's cache. Need to pontificate about how Google is potentially evil but can't keep your server running? Don't worry, people can read it via Google's cache.
- pyre 14y agoWhy don't people just post things via a Coral Cache link when it's not a major site (that should obviously be able to handle the traffic)?
- username111 14y agoBecause it isn't proper etiquette, it is their content and they should get the views for it. Some people may not be able to handle it and caches are always nice but pageviews are also a nice thing when you run a website even if they don't generate any revenue.
- pyre 14y agoOk. True enough. That said, having a Coral Cache link auto-generated for each article[1] has been a requested feature of HN for a while. E.g. 1. Article Title (example.com) [Coral Cache Link] That way it can be cached prior to going down. [1] At least, each article that hits the front page.
- perlgeek 14y agoI just tried the digg.com/robots.txt, and for it me it says: User-agent: * Disallow: Which explains everything without conspiracy theories
- philip1209 14y agoYes, this seems to explain it. http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html
- dredge 14y agoNo, it doesn't. Your linked example contains: User-agent: * Disallow: / While the (new) digg.com/robots.txt contains: User-agent: * Disallow: Those are very different. The former essentially disallows bots from crawling the entire site, while the latter disallows nothing - effectively allowing everything. The syntax is unusual, granted, for historical reasons.
- ecaron 14y agoA "Disallow:" (without anything after it) is saying allow all robots access to the entire server. (If you need something beyond my word on the topic, there's a thorough explanation at http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html)