3 ms·

The article is a bit wrong. nofollow does not mean that the crawlers must not follow the link, but means that the ranking algorithms must not consider the link

by Loic 15y ago

The article is a bit wrong. nofollow does not mean that the crawlers must not follow the link, but means that the ranking algorithms must not consider the link for the ranking as the link could be dubious or spam. As such, the relationship between the page linked and the page linking is not to be affected by the link. And this is what makes the links not surfacing. Basically, the current Twitter HTML says: We have no outbound links.

chalst 15y ago
Google has said that it won't add a URL to its index of URLs to crawl as a result of seeing the URL in a rel=nofollow href [1]. This is over and above not weighting the link if the URL does otherwise make it onto its to crawl list. [1] http://googleblog.blogspot.com/2007/02/robots-exclusion-protocol.html http://googleblog.blogspot.com/2007/02/robots-exclusion-prot...
- Loic 15y ago
  Sure, but the probability of having an URL shared only through a tweet is zero. Google is insanely efficient at finding new URLs. So, it will find it anyway, but it will not be shown to you as "shared by your friend John" because of the nofollow.
  - Terretta 15y ago
    > the probability of having an URL shared only through a tweet is zero Countless URLs are shared only through tweets, for example URLs that are private URLs to content on services such as TwitPic[1]. 1. http://www.dailymail.co.uk/tvshowbiz/article-2084361/AnnaLynne-McCord-topless-Twitter-picture-90210-star-accidentally-posts-nude-photo.html http://www.dailymail.co.uk/tvshowbiz/article-2084361/AnnaLyn... (NSFW)
- dolbz 15y ago
  The post you're linking is discussing the META tag version of nofollow, not the hyperlink rel attribute which is detailed here: http://googleblog.blogspot.com/2005/01/preventing-comment-spam.html http://googleblog.blogspot.com/2005/01/preventing-comment-sp... This article doesn't say explicitly that it won't follow those links and I would suggest it often will as comments are a rich source of pages to index. The nofollow on a link like this just indicates that the page owner can't vouch for the quality of the linked page (e.g. user submitted link) and therefore doesn't want to pass pagerank to it. Edit: it's also implied in the parent's link that the META tag only prevents the following of links to pages within your site. This sounds reasonable as who are you to tell Google they can't index a third party domain?
  - mapgrep 15y ago
    "who are you to tell Google they can't index a third party domain?" No tweets link to third-party domains, since Twitter wraps all links in their own URL shortener (t.co). Even if you use an independent URL shortener, it will be wrapped by t.co. (I suppose you could argue that there is no way for Google to determine this algorithmically, since "twitter.com" != "t.co", so it should go ahead with the crawl, but there's the question of how Twitter would respond to that.)
    - Raphael 15y ago
      They could create a robots.txt for t.co and set it to noindex.
- gojomo 15y ago
  Note when Google first unveiled their 'rel=nofollow' invention in 2005, they were very careful to say only that it meant 'no weighting', but not necessarily 'no discovery via this reference'. The attempt to formalize the attribute maintained that same distinction. [1] The Google manager blogging about it two years later (five years ago) was probably correct about then-current practice, but I wouldn't count on that as an enduring commitment that Google had never and would never follow such links. [1] http://microformats.org/wiki/rel-nofollow http://microformats.org/wiki/rel-nofollow
luigi 15y ago
It's true that there's a difference between crawling, indexing, and ranking, but Google is pretty clear on the ultimate effect. http://support.google.com/webmasters/bin/answer.py?hl=en&answer=96569 http://support.google.com/webmasters/bin/answer.py?hl=en&#38... How does Google handle nofollowed links? In general, we don't follow them. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from our overall graph of the web. However, the target pages may still appear in our index if other sites link to them without using nofollow, or if the URLs are submitted to Google in a Sitemap.
- masklinn 15y ago
  That's the robots nofollow, we're talking about a/@rel=nofollow which google advertises quite differently: http://googleblog.blogspot.com/2005/01/preventing-comment-spam.html http://googleblog.blogspot.com/2005/01/preventing-comment-sp... and the microformats standardization[0] stuck very carefully to "provide no additional weight or ranking to target" period. Not that crawlers can't follow that link for new content, and not that crawlers can't reverse that link for "shared on" features, just that this precise link won't add to the pagerank of the target. [0] http://microformats.org/wiki/rel-nofollow http://microformats.org/wiki/rel-nofollow
  - jasonlotito 15y ago
    Dear god, your basically repeating one link from 7 years ago. Provided here as information for people who might think you are offering a valid resource: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=96569 http://support.google.com/webmasters/bin/answer.py?hl=en&#38...
  - shadowfiend 15y ago
    The thing is, if Google were to say “x was shared by y on twitter”, that is precisely providing some additional weight to the result. Human weight, admittedly, though I believe this also factors into the weight on the page. So it makes sense that twitter links would also not be included in this sharing stuff either.

by Loic 15y ago

chalst 15y ago

Google has said that it won't add a URL to its index of URLs to crawl as a result of seeing the URL in a rel=nofollow href [1]. This is over and above not weighting the link if the URL does otherwise make it onto its to crawl list. [1] http://googleblog.blogspot.com/2007/02/robots-exclusion-protocol.html http://googleblog.blogspot.com/2007/02/robots-exclusion-prot...

Loic 15y ago
Sure, but the probability of having an URL shared only through a tweet is zero. Google is insanely efficient at finding new URLs. So, it will find it anyway, but it will not be shown to you as "shared by your friend John" because of the nofollow.
- Terretta 15y ago
  > the probability of having an URL shared only through a tweet is zero Countless URLs are shared only through tweets, for example URLs that are private URLs to content on services such as TwitPic[1]. 1. http://www.dailymail.co.uk/tvshowbiz/article-2084361/AnnaLynne-McCord-topless-Twitter-picture-90210-star-accidentally-posts-nude-photo.html http://www.dailymail.co.uk/tvshowbiz/article-2084361/AnnaLyn... (NSFW)
dolbz 15y ago
The post you're linking is discussing the META tag version of nofollow, not the hyperlink rel attribute which is detailed here: http://googleblog.blogspot.com/2005/01/preventing-comment-spam.html http://googleblog.blogspot.com/2005/01/preventing-comment-sp... This article doesn't say explicitly that it won't follow those links and I would suggest it often will as comments are a rich source of pages to index. The nofollow on a link like this just indicates that the page owner can't vouch for the quality of the linked page (e.g. user submitted link) and therefore doesn't want to pass pagerank to it. Edit: it's also implied in the parent's link that the META tag only prevents the following of links to pages within your site. This sounds reasonable as who are you to tell Google they can't index a third party domain?
- mapgrep 15y ago
  "who are you to tell Google they can't index a third party domain?" No tweets link to third-party domains, since Twitter wraps all links in their own URL shortener (t.co). Even if you use an independent URL shortener, it will be wrapped by t.co. (I suppose you could argue that there is no way for Google to determine this algorithmically, since "twitter.com" != "t.co", so it should go ahead with the crawl, but there's the question of how Twitter would respond to that.)
  - Raphael 15y ago
    They could create a robots.txt for t.co and set it to noindex.
gojomo 15y ago
Note when Google first unveiled their 'rel=nofollow' invention in 2005, they were very careful to say only that it meant 'no weighting', but not necessarily 'no discovery via this reference'. The attempt to formalize the attribute maintained that same distinction. [1] The Google manager blogging about it two years later (five years ago) was probably correct about then-current practice, but I wouldn't count on that as an enduring commitment that Google had never and would never follow such links. [1] http://microformats.org/wiki/rel-nofollow http://microformats.org/wiki/rel-nofollow

luigi 15y ago

It's true that there's a difference between crawling, indexing, and ranking, but Google is pretty clear on the ultimate effect. http://support.google.com/webmasters/bin/answer.py?hl=en&answer=96569 http://support.google.com/webmasters/bin/answer.py?hl=en&#38... How does Google handle nofollowed links? In general, we don't follow them. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from our overall graph of the web. However, the target pages may still appear in our index if other sites link to them without using nofollow, or if the URLs are submitted to Google in a Sitemap.

masklinn 15y ago
That's the robots nofollow, we're talking about a/@rel=nofollow which google advertises quite differently: http://googleblog.blogspot.com/2005/01/preventing-comment-spam.html http://googleblog.blogspot.com/2005/01/preventing-comment-sp... and the microformats standardization[0] stuck very carefully to "provide no additional weight or ranking to target" period. Not that crawlers can't follow that link for new content, and not that crawlers can't reverse that link for "shared on" features, just that this precise link won't add to the pagerank of the target. [0] http://microformats.org/wiki/rel-nofollow http://microformats.org/wiki/rel-nofollow
- jasonlotito 15y ago
  Dear god, your basically repeating one link from 7 years ago. Provided here as information for people who might think you are offering a valid resource: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=96569 http://support.google.com/webmasters/bin/answer.py?hl=en&#38...
- shadowfiend 15y ago
  The thing is, if Google were to say “x was shared by y on twitter”, that is precisely providing some additional weight to the result. Human weight, admittedly, though I believe this also factors into the weight on the page. So it makes sense that twitter links would also not be included in this sharing stuff either.