8 ms·

Stackoverflow abuses nofollow (2011)

126 points by nerfhammer 14y ago

HPBEggo 14y ago
I can understand why they do it - diminishing the page rank of sites that give answers to questions on Stack Overflow causes Stack Overflow to gain more traffic - but it still seems kind of underhanded. They rely on others for the content of the site, so they should give credit to the producers of that content - without it, they would have no service at all.
debacle 14y ago
This is an interesting reality. I know that some of the SO guys frequent HN. I wonder if we'll see a valid response. The only possible reason I could see is spam avoidance.
sologoub 14y ago
Although I get the concern here, it would seem that if you removed the nofollows, it would open up the site to all kinds of abuse and diminish the value of the content very quickly. It doesn't strike me as a very difficult task to create a few thousand profiles and have them vote for each other (and also give you a ton of votes to random people to hide better), and then use the collected power for SEO spam. I agree with you that there has to be a better solution, but it doesn't strike me as a very trivial one... any thoughts on how one would approach this?
- streptomycin 14y ago
  The article seems to argue that because there is no perfect solution to spam (i.e. spammers will still want their links on SO even if they are nofollowed), then there is no point in marginally reducing the incentive to spam.
  - cheald 14y ago
    I think the point is myopic. Human moderators will eventually clean up spam, but Google is freakishly fast at indexing things like StackOverflow. Omitting rel=nofollow would mean that SO ends up lending linkjuice to spammers at least some of the time, thereby providing spammers with incentive to relentless spam even if their spam is cleaned up by humans. My general rule of thumb is "any content that comes from untrusted users gets a nofollow stamp". It's "XSS prevention" for your pagerank.
    - bryanlarsen 14y ago
      But SO knows which users are trusted, and which aren't. So why is it adding rel=nofollow to trusted users?
      - cheald 14y ago
        When I say "trusted users", I mean "people who have the product's interests before their own". I don't think this applies to users, even high-reptuation users. Arguably, it eliminates incentive to link to anything except the most useful resource. I have >11k reputation on SO, and I'll easily admit that I'd be tempted to link to my blog or whatnot for an answer whenever I could wedge it in, if I could siphon off some of SO's linkjuice. I'm not interested in spamming or anything, but that's how the game is played; SO has such a massive amount of clout with Google that any non-nofollow links from it confer an awful lot of weight. By just nofollowing everything, they disincentivize people from even being tempted to game the system and promote their own stuff, even if it were done in a helpful way. Instead, I am incentivized to provide the most useful, comprehensive link I can find to answer a question. All my incentive is in earning an acceptance of my answer, rather than just in having an answer good enough to not not get spam-binned that sits there and feeds me pagerank.
      - lambda 14y ago
        I have seen reputable open source software developers put paid linkspam on their personal websites. When I called them out about it, they said "ha ha, I was wondering if anyone would notice that, yeah, they paid me a pretty good amount of money for that." Trusted contributors could easily be bought off. Someone who was sick and tired of StackOverflow after putting way too much time into it, who was offered the right amount of money? Hey, you know, it might be worth it to make a few bucks instead of a few more meaningless internet points. If you just make the threshold so high that only the very top contributors are nofollowed, then only that small cabal controls where SO's considerable linkjuice goes. If you broaden your criteria, there are more and more potential corruptible people; and it becomes easier to generate a little quick and easy reputation with a sockpuppet that you can spam with a few times before you're caught and move on to another sockpuppet.
- zalambar 14y ago
  As the article mentions; at a certain reputation level StackExchange users are granted a followed link in their profile. Similar metrics could be applied to questions and answers with the "nofollow" being removed from highly rated user content which is unlikely to be spam. Still a hard problem but author reputation and question/answer votes give a good set of data from which we might identify links to sources which deserve attribution.
- greenyoda 14y ago
  "it would seem that if you removed the nofollows, it would open up the site to all kinds of abuse" That's why the author of the article suggested that they remove "nofollow" for users who have an established reputation, like SlashDot does. That algorithm would seem to be very easy to implement.
  - manojlds 14y ago
    What happens when a less established user edits the post of a more established one ( and maybe also vice versa)
    - bryanlarsen 14y ago
      You need a good amount of karma to edit someone else's posts, so that's a non-issue. If it was an issue, the rel=nofollow should be based on the rep of the user creating the link, whether that was the poster or the editor.
      - jakub_g 14y ago
        You need 2000 rep to do this, which is not so easy to gain, but certainly achievable for a highly determined person. However, what @wookietrader wrote applies -- each edit is marked with the person who has done it which solves the problem.
      - manojlds 14y ago
        Anyone can edit. If your rep is low, it has to be approved.
    - wookietrader 14y ago
      Since you can check what user added what link (using diff), you can still use the reputation threshold. If that does not scale, just use the minimum reputation of the editors' reputations.
  - nerfhammer 14y ago
    Interestingly they already remove nofollow from links in your profile after you reach a certain karma threshold
- bryanlarsen 14y ago
  I'm confident that stackoverflow already utilizes voting ring detection logic. High karma on SO is quite valuable, I'm sure there are many that have already tried to game the system, people who are probably a lot better at evading the voting ring detection than the average spammer...
  - dsl 14y ago
    I'm curious how karma on SO is valuable. Not in the "you seem smart i'll give you a job" sense of value, but in the actual converted into dollars value that spammers go after.
    - bryanlarsen 14y ago
      It's valuable in the "I'm a leet hacker" sense.
- jere 14y ago
  >It doesn't strike me as a very difficult task to create a few thousand profiles and have them vote for each other (and also give you a ton of votes to random people to hide better), and then use the collected power for SEO spam. I have to disagree. I bet this kind of abuse is well studied by any major site relying on a reputation system/user content. I can't point to anything specific, but off the top of my head discovering that kind of abuse seems exactly like finding strongly connected components: http://en.wikipedia.org/wiki/Strongly_connected_component http://en.wikipedia.org/wiki/Strongly_connected_component So you could probably prevent this algorithmically, but even if you couldn't... the site is heavily moderated. Bad questions and answers are down voted or closed. In my experience, getting lots of reputation on SO actually requires a lot of persistent effort. I have just over 1000 rep myself and it felt like it took forever to get there. Even if you could get lots of upvotes, you'd still be looking at daily reputation limits. And finally, you'd waste all that on one or two spam links only to get perma-banned and have all your links deleted? A combination of high reputation + an initial nofollow time period (to allow spam links to be discovered) seems like it would be pretty effective. There's really only one route to taking advantage of the google juice: http://xkcd.com/810/ http://xkcd.com/810/
  - sologoub 14y ago
    While I agree that it would be possible to get these networks exposed, it is still non-trivial effort. That's really the point I am making. Personally, I'd rather see them allocate more resources towards making developer QA even better, than chasing protective features. My 1781 reputation was not that hard to come by - it happened organically with me trying to help out in the Google App Engine section. I'm not a developer by trade, just a hobbyist. Based on that experience, I'd imagine there to be areas that are much more obscure, with little moderation oversight, being decent sources of easy reputation points...
gojomo 14y ago
Sites are free to overuse 'rel=nofollow', and search engines are free to selectively ignore it (for either link-discovery or ranking purposes). The strongest point I see made by this author is about SO's hypocrisy: SO requires attribution to be with a link, and specify that link must not have 'rel=nofollow'. Yet their content relies heavily on references to elsewhere which are all 'rel=nofollow'ed. A sense of fair play in attribution, and spirit of mutual assistance between reliable authorities, would suggest allowing at least some well-vetted outlinks to be unencumbered.
- nerfhammer 14y ago
  Quoting Jeff Atwood: If you republish this content, we require that you: [...] Hyperlink directly to the original question on the source site [...] By “directly”, I mean each hyperlink must point directly to our domain in standard HTML visible even with JavaScript disabled, and not use a tinyurl or any other form of obfuscation or redirection. Furthermore, the links must not be nofollowed. This is about the spirit of fair attribution. Attribution to the website, and more importantly, to the individuals who so generously contributed their time to create that content in the first place! Anyway, I hope that clears up any confusion — feel free to remix and reuse to your heart’s content, as long as a good faith effort is made to attribute the content! http://blog.stackoverflow.com/2009/06/attribution-required/ http://blog.stackoverflow.com/2009/06/attribution-required/
- esrauch 14y ago
  If the link to stackexchange was user-generated content I seriously doubt they would actually expect it to have nofollow. The outgoing links to any site from the stackexchange blog do not have nofollow. Any content that they are explicitly curating does not have nofollow, any user content does. I think that is a reasonably consistent policy.
SenorWilson 14y ago
They're technically using it for it's purpose, or they can claim they are even if they have a different motive. Stackoverflow allows links in comments and posts and this could be abused.
- eli 14y ago
  Sites with high pagerank that allow user submitted links are few and far between. If they didn't put nofollow (at least for new accounts), SO is popular enough that it would have to deal with targeted spam attacks aimed specifically at them. That is a sucky problem to deal with.
- walshemj 14y ago
  Will be abused sadly ;-(
theseanstewart 14y ago
I feel the exact same way about Wikipedia.
- patmcguire 14y ago
  Especially since the revert brigade makes it very difficult to make any legitimate changes, let alone spammy ones, on any page that would be worth it.
eli 14y ago
I guess I'm in the minority, but I'm not really buying that this is abusive. Yes, it would be nice if SO removed the nofollow from known-good posts (which, indeed, it seems they are now doing), but adding nofollow is a pretty simple and reasonable way to make the site a whole lot less attractive to some of the most obnoxious sort of spammers. I would hope nobody is adding links to their SO answers with the expectation of an SEO benefit.
- bryanlarsen 14y ago
  If there's a link in a stackoverflow answer of mine, it's there because there's really good information at the end of the link. That's the sort of information Google & Bing use to make their searches useful, it's the sort of information I want Google et al to know.
  - drivebyacct2 14y ago
    Yes, I agree. StackOverflow is harming the quality of results in Google because they want people to filter through them for the information. Anyone that's used SO knows that answers can be misleading, incomplete or straight up misinformed. At the very least they're generally less comprehensive and less keyword-full than source material that would appear as links. I suppose it's selfish of me to want SO to share them and selfish of SO to mark them as nofollow and not allow the source articles to rise in ranking.
    - vitalique 14y ago
      Original topic is not explicitly about content, but speaking of SO's content: > StackOverflow is harming the quality of results in Google because they want people to filter through them for the information. Well, no, actually. StackOverflow may be shooting itself in the foot. SO cannot dictate the rules of the search game to Google, and Google adjusts its SERPs pretty quickly, especially nowadays, when user behavior gets to have bigger influence on SERP with every ranking algo update. Poor quality content leads to higher bounce rate and lower avg. time on site and visit depth, which will inevitably lead to lower ranking and less traffic for SO. This (and perhaps a thousand of other factors) works for every site, and even more so for heavy traffic content projects like StackOverflow. So don't be afraid, the system will adjust itself. Overall, I agree with your sentiment: I love SO, but nofollowing the Web looks kind of lame and selfish. After all, doesn't SO have its staff of moderators and admins to fight spammy comments?
jeremysmyth 14y ago
This topic is discussed quite frequently in the meta pages on StackOverflow. For example, http://meta.stackoverflow.com/questions/111279/remove-nofollow-on-links-deemed-reputable http://meta.stackoverflow.com/questions/111279/remove-nofoll... covers exactly this subject. SO gets lots of googlejuice, and because it's gameable, they appear to want to give some of that googlejuice only to "reputable" links (which they do for links in high-rep users' profiles). That's a really hard thing to judge, but the fact that they're thinking about it, talking about it, and soliciting feedback on it means that they're trying to do the right thing, even if the article's author doesn't think their "right thing" is good enough for him.
VMG 14y ago
The author doesn't address this response by Jeff Atwood: URGENT UPDATE We were seeing a significant drop in Google (organic) traffic for Server Fault after instituting this "follow links if enough upvotes post-edit or post-create" policy. We traced it back to what we currently think are a string of posts on Server Fault that got nofollow removed through "trust", but were being interpreted by Google as link farms or spammy pages. [....] http://meta.stackoverflow.com/a/51156 http://meta.stackoverflow.com/a/51156
- bryanlarsen 14y ago
  There's a follow-up to the URGENT UPDATE on SO: http://meta.stackoverflow.com/questions/111279/remove-nofollow-on-links-deemed-reputable http://meta.stackoverflow.com/questions/111279/remove-nofoll... So it appears that they do remove rel=nofollow from reputable links, although their threshold for reputable appears to be very high.
- preinheimer 14y ago
  How is that not the expected result? When the sites that informed various well regarded answers on the sites get their due, their search engine rank rises. With such a well ranked site now pointing towards them, they rise in the results, often higher than the SO answer referencing it. As a secondary issue, there's a relatively common black-hat-seo practice of buying a well regarded domain, and soaking that links for all they're worth to promote something. Their switch-flipping may have run afoul of systems designed to mitigate that.
- kamjam 14y ago
  The author posted the original article on 17th Dec 2010 and an update on 11th Dec 2011. So not sure how this made it to the front page of HN. Not surprised the author doesn't address the response.
cstrat 14y ago
Someone have a mirror of the article?
- cstrat 14y ago
  google cache seems ok: http://webcache.googleusercontent.com/search?q=cache:3sKnQOnbA0UJ:www.brianbondy.com/blog/id/104/stackoverflow-amongst-nofollow-web-abuse-sites+&cd=1&hl=en&ct=clnk&gl=au http://webcache.googleusercontent.com/search?q=cache:3sKnQOn...
kmontrose 14y ago
This was dealt with more than a year ago, when we did remove nofollow from trusted posts links' http://meta.stackoverflow.com/questions/111279/remove-nofollow-on-links-deemed-reputable http://meta.stackoverflow.com/questions/111279/remove-nofoll.... Would have been dealt with even earlier, but our experiments with it (http://meta.stackoverflow.com/a/69032/130213 http://meta.stackoverflow.com/a/69032/130213) lead us to believe Google was starting to treat us as a link farm. What we were seeing in our earlier attempt wasn't pages appearing lower in results (which could reasonably be expected in some cases as a result of the change; and we wouldn't care, 2nd on a search under an equally good resource is fine by us), but pages being removed entirely; almost universally pages that had newly un-nofollow-ed links to stores, or ad-laden sites (mostly legitimate posts, but some spam that stuck around for a bit before the community deleted it). So, "classified as link farm" seemed like the most likely problem. Google naturally won't tell you why your ranking drops, so it was (and remains) just an educated guess as to what happened. So we stopped, putting it on the "wish we could, but reality doesn't let us"-list until Google reached out to us (among thousands of others, I'm sure) to change our nofollow practices. Google didn't describe any changes in their algorithm, but it seems reasonable to me that there would have been some tweaks around nofollow to accompany a new policy; again just an educated guess. Basically, this is an old post complaining about a long since addressed concern that we had tried to address even earlier but ran into practical problems with. While the exact details of our algorithm are secret by necessity, I will say that we've had to consider posts individually to prevent spammers from kiting a single account up to post nofollow-less spam content. People still try, it's kind of astounding how much spammers try (I suspect SEO's opaqueness cuts both ways here), but it doesn't work (well, you can get one link in your profile; but there's less SEO juice to pass and you are hard-capped at one, no matter how long it takes someone to delete your account). Disclaimer: Stack Exchange employee, I was on all the relevant calls but has been a couple years so grain of salt and all that.
- nikatwork 14y ago
  I've seen this happen to other sites, something angers the Google algo and organic traffic falls off a cliff. History seems to bear out that SO correctly identified the source of the "problem", and SO is such a valuable resource that I'd rather it be Google-visible over passing pagerank. This is the problem when you have a single player dominating the search space with an opaque algo and a very limited appeals process.
bhanks 14y ago
How is this news? 1. This post is from 2011 2. Google ended the ability to link sculpt a long time ago. I would be interested in studies showing cases where link sculpting still worked.
- kamjam 14y ago
  Agree. The original post is actually from from Dec 2010 and an update on Dec 2011!
robomartin 14y ago
I really don't see SO as being the problem here. I happen to think that secret SEO/page rank algorithms are the culprit in many ways. Google is in the unique position as to be able to write a "how to behave" guide for the internet. Play by their rules and you are golden, start to violate them and you start to be "punished" in a gradual (but exponentially more severe). The whole time full feedback and data on what you are doing wrong is provided. As this approach evolves spammers would be less and less successful and might just have to resort to legitimate means of marketing.
- JangoSteve 14y ago
  Furthermore, the search engines could nip this kind of abuse in the bud, by ignoring the "nofollow" attribute of the links on a site if it determines that site is abusing it (e.g. if all or some high percentage of the links on the site have "nofollow").
pootch 14y ago
SO has serious content problems of which nofollow is not one of them. SO cant decide if it wants deep content or shallow content, and it has mostly ended up being shallow (and massive amounts of incorrect deep/shallow content), links outside of SO are a major PAIN, as SO is in the format business not the content business. If the "answer" is not in SOs format which sometimes barely ranks above commenting, then its a bad thing. If you have blogworthy content dont bother posting it on SO, its a waste of time and in the Q&A format of SO its also a nuisance.
netcan 14y ago
This nofollow stuff has always seemed very weird to me for several reasons. - Search Engines are supposed to be analyzing the web, figuring out whats important. They have the incentive to do this well. The websites they are analyzing do not. Websites have an incentive to nofollow everything. They might be worried about their own rankings, but why should they care about the sites they link to. Nofollow is safe and harmless. No-nofolow should get you in trouble. what does a site gain from not using nofollow links on everything? - To enforce nofollow rules, Google are supposedly disciplining sites by hurting their organic rankings. But a page with nofollow links appears exactly the same to a user. It's just as right an answer as it was before. Is Google lowering the quality of its results to police the web? - If Google are able to detect "link farms" and user generated content that should have been nofollow-ed, why don't they just treat those links as nofollow and ignore them. Use this detection to analyze the web, not police it. - Are they ignoring important information sources? A huge, information rich portion of the web is user generated. Eg wikipedia & stackoverflow. How can Google really be ignoring these links here as data sources? The links on a wikipedia page for example, are very informative. If some webpage is mentioned frequently in stackoverflow questions and answers, it's probably important and its probably a good answer to a lot of questions people are asking google.
doyouevenlift 14y ago
No one cares.
205guy 14y ago
Esprit d'escalier: "In my day (usenet), we didn't have links--and we liked it." (old SNL joke for those who don't get it). This conversation has 2 angles I find very interesting. The first is the whole question of quality of the content. Google is a huge AI trying to give you the same quality of information that you'd select if you looked at the same (millions of) sources. At first, I liked netcan's answer[1], which says essentially, nofollow them all or none, Google will sort out its own. But the more I think about it, the more context that sites can provide automatically, the better. If a site can say "this content provided by untrusted source" (because we're a community-driven site, and we can't police everything), that's a help to Google's Algorithm. Google is free to still follow the link and perhaps, using it's other knowledge of the target, assign a trustworthiness score to the nofollow quality on the original site. If a good community is providing good links, but the website owner still marks them nofollow, that might become a badge of honor in Google eyes (all-crawling robot tentacles?). As a side note, I love how everybody, from slimy SEO to internet ethicist, is trying to guess Google's secret formula (it used to be Coca-Cola that had a secret formula that was releveant). Google is essentially the unknowable proto-God of this new information universe. The other interesting topic is getting paid for content. My first reaction to the OP was "wah, wah, OP doesn't want to pay for his little corner of the internet." Let's face it, the internet doesn't run for free (hardware and electricity), and the content that attracts people doesn't write itself. StackOverflow provides a community for people to deliver their content into and receive socaial rewards. If SO can't do that because of some quirk of SEO, then it's likely it wouldn't be able to host the community anymore. The internet has still not figured out a way to pay for that other than advertizing (the forcing of unwanted content on readers). But advertizing only works if you can keep track of views and stay on top of the SEO game. I totally agree they're hypocrites for hoarding the follows (must follow to us be we nofollow to everyone), but how's that different from a corporate board's fiduciary duty to shareholders to maximize the profit of the company? I also think cheald[2] makes a very good point about why the dynamics must be towards nofollow. The question is then how involved or committed are you to the community, and how do you feel rewarded for the time you contribute to it. [1] http://news.ycombinator.com/item?id=4777260 http://news.ycombinator.com/item?id=4777260 [2] http://news.ycombinator.com/item?id=4774888 http://news.ycombinator.com/item?id=4774888

126 points by nerfhammer 14y ago

I can understand why they do it - diminishing the page rank of sites that give answers to questions on Stack Overflow causes Stack Overflow to gain more traffic - but it still seems kind of underhanded. They rely on others for the content of the site, so they should give credit to the producers of that content - without it, they would have no service at all.

debacle 14y ago

This is an interesting reality. I know that some of the SO guys frequent HN. I wonder if we'll see a valid response. The only possible reason I could see is spam avoidance.

sologoub 14y ago

Although I get the concern here, it would seem that if you removed the nofollows, it would open up the site to all kinds of abuse and diminish the value of the content very quickly. It doesn't strike me as a very difficult task to create a few thousand profiles and have them vote for each other (and also give you a ton of votes to random people to hide better), and then use the collected power for SEO spam. I agree with you that there has to be a better solution, but it doesn't strike me as a very trivial one... any thoughts on how one would approach this?

streptomycin 14y ago
The article seems to argue that because there is no perfect solution to spam (i.e. spammers will still want their links on SO even if they are nofollowed), then there is no point in marginally reducing the incentive to spam.
- cheald 14y ago
  I think the point is myopic. Human moderators will eventually clean up spam, but Google is freakishly fast at indexing things like StackOverflow. Omitting rel=nofollow would mean that SO ends up lending linkjuice to spammers at least some of the time, thereby providing spammers with incentive to relentless spam even if their spam is cleaned up by humans. My general rule of thumb is "any content that comes from untrusted users gets a nofollow stamp". It's "XSS prevention" for your pagerank.
  - bryanlarsen 14y ago
    But SO knows which users are trusted, and which aren't. So why is it adding rel=nofollow to trusted users?
    - cheald 14y ago
      When I say "trusted users", I mean "people who have the product's interests before their own". I don't think this applies to users, even high-reptuation users. Arguably, it eliminates incentive to link to anything except the most useful resource. I have >11k reputation on SO, and I'll easily admit that I'd be tempted to link to my blog or whatnot for an answer whenever I could wedge it in, if I could siphon off some of SO's linkjuice. I'm not interested in spamming or anything, but that's how the game is played; SO has such a massive amount of clout with Google that any non-nofollow links from it confer an awful lot of weight. By just nofollowing everything, they disincentivize people from even being tempted to game the system and promote their own stuff, even if it were done in a helpful way. Instead, I am incentivized to provide the most useful, comprehensive link I can find to answer a question. All my incentive is in earning an acceptance of my answer, rather than just in having an answer good enough to not not get spam-binned that sits there and feeds me pagerank.
    - lambda 14y ago
      I have seen reputable open source software developers put paid linkspam on their personal websites. When I called them out about it, they said "ha ha, I was wondering if anyone would notice that, yeah, they paid me a pretty good amount of money for that." Trusted contributors could easily be bought off. Someone who was sick and tired of StackOverflow after putting way too much time into it, who was offered the right amount of money? Hey, you know, it might be worth it to make a few bucks instead of a few more meaningless internet points. If you just make the threshold so high that only the very top contributors are nofollowed, then only that small cabal controls where SO's considerable linkjuice goes. If you broaden your criteria, there are more and more potential corruptible people; and it becomes easier to generate a little quick and easy reputation with a sockpuppet that you can spam with a few times before you're caught and move on to another sockpuppet.
zalambar 14y ago
As the article mentions; at a certain reputation level StackExchange users are granted a followed link in their profile. Similar metrics could be applied to questions and answers with the "nofollow" being removed from highly rated user content which is unlikely to be spam. Still a hard problem but author reputation and question/answer votes give a good set of data from which we might identify links to sources which deserve attribution.
greenyoda 14y ago
"it would seem that if you removed the nofollows, it would open up the site to all kinds of abuse" That's why the author of the article suggested that they remove "nofollow" for users who have an established reputation, like SlashDot does. That algorithm would seem to be very easy to implement.
- manojlds 14y ago
  What happens when a less established user edits the post of a more established one ( and maybe also vice versa)
  - bryanlarsen 14y ago
    You need a good amount of karma to edit someone else's posts, so that's a non-issue. If it was an issue, the rel=nofollow should be based on the rep of the user creating the link, whether that was the poster or the editor.
    - jakub_g 14y ago
      You need 2000 rep to do this, which is not so easy to gain, but certainly achievable for a highly determined person. However, what @wookietrader wrote applies -- each edit is marked with the person who has done it which solves the problem.
    - manojlds 14y ago
      Anyone can edit. If your rep is low, it has to be approved.
  - wookietrader 14y ago
    Since you can check what user added what link (using diff), you can still use the reputation threshold. If that does not scale, just use the minimum reputation of the editors' reputations.
- nerfhammer 14y ago
  Interestingly they already remove nofollow from links in your profile after you reach a certain karma threshold
bryanlarsen 14y ago
I'm confident that stackoverflow already utilizes voting ring detection logic. High karma on SO is quite valuable, I'm sure there are many that have already tried to game the system, people who are probably a lot better at evading the voting ring detection than the average spammer...
- dsl 14y ago
  I'm curious how karma on SO is valuable. Not in the "you seem smart i'll give you a job" sense of value, but in the actual converted into dollars value that spammers go after.
  - bryanlarsen 14y ago
    It's valuable in the "I'm a leet hacker" sense.
jere 14y ago
>It doesn't strike me as a very difficult task to create a few thousand profiles and have them vote for each other (and also give you a ton of votes to random people to hide better), and then use the collected power for SEO spam. I have to disagree. I bet this kind of abuse is well studied by any major site relying on a reputation system/user content. I can't point to anything specific, but off the top of my head discovering that kind of abuse seems exactly like finding strongly connected components: http://en.wikipedia.org/wiki/Strongly_connected_component http://en.wikipedia.org/wiki/Strongly_connected_component So you could probably prevent this algorithmically, but even if you couldn't... the site is heavily moderated. Bad questions and answers are down voted or closed. In my experience, getting lots of reputation on SO actually requires a lot of persistent effort. I have just over 1000 rep myself and it felt like it took forever to get there. Even if you could get lots of upvotes, you'd still be looking at daily reputation limits. And finally, you'd waste all that on one or two spam links only to get perma-banned and have all your links deleted? A combination of high reputation + an initial nofollow time period (to allow spam links to be discovered) seems like it would be pretty effective. There's really only one route to taking advantage of the google juice: http://xkcd.com/810/ http://xkcd.com/810/
- sologoub 14y ago
  While I agree that it would be possible to get these networks exposed, it is still non-trivial effort. That's really the point I am making. Personally, I'd rather see them allocate more resources towards making developer QA even better, than chasing protective features. My 1781 reputation was not that hard to come by - it happened organically with me trying to help out in the Google App Engine section. I'm not a developer by trade, just a hobbyist. Based on that experience, I'd imagine there to be areas that are much more obscure, with little moderation oversight, being decent sources of easy reputation points...

gojomo 14y ago

Sites are free to overuse 'rel=nofollow', and search engines are free to selectively ignore it (for either link-discovery or ranking purposes). The strongest point I see made by this author is about SO's hypocrisy: SO requires attribution to be with a link, and specify that link must not have 'rel=nofollow'. Yet their content relies heavily on references to elsewhere which are all 'rel=nofollow'ed. A sense of fair play in attribution, and spirit of mutual assistance between reliable authorities, would suggest allowing at least some well-vetted outlinks to be unencumbered.

nerfhammer 14y ago
Quoting Jeff Atwood: If you republish this content, we require that you: [...] Hyperlink directly to the original question on the source site [...] By “directly”, I mean each hyperlink must point directly to our domain in standard HTML visible even with JavaScript disabled, and not use a tinyurl or any other form of obfuscation or redirection. Furthermore, the links must not be nofollowed. This is about the spirit of fair attribution. Attribution to the website, and more importantly, to the individuals who so generously contributed their time to create that content in the first place! Anyway, I hope that clears up any confusion — feel free to remix and reuse to your heart’s content, as long as a good faith effort is made to attribute the content! http://blog.stackoverflow.com/2009/06/attribution-required/ http://blog.stackoverflow.com/2009/06/attribution-required/
esrauch 14y ago
If the link to stackexchange was user-generated content I seriously doubt they would actually expect it to have nofollow. The outgoing links to any site from the stackexchange blog do not have nofollow. Any content that they are explicitly curating does not have nofollow, any user content does. I think that is a reasonably consistent policy.

SenorWilson 14y ago

They're technically using it for it's purpose, or they can claim they are even if they have a different motive. Stackoverflow allows links in comments and posts and this could be abused.

eli 14y ago
Sites with high pagerank that allow user submitted links are few and far between. If they didn't put nofollow (at least for new accounts), SO is popular enough that it would have to deal with targeted spam attacks aimed specifically at them. That is a sucky problem to deal with.
walshemj 14y ago
Will be abused sadly ;-(

theseanstewart 14y ago

I feel the exact same way about Wikipedia.

patmcguire 14y ago
Especially since the revert brigade makes it very difficult to make any legitimate changes, let alone spammy ones, on any page that would be worth it.

eli 14y ago

I guess I'm in the minority, but I'm not really buying that this is abusive. Yes, it would be nice if SO removed the nofollow from known-good posts (which, indeed, it seems they are now doing), but adding nofollow is a pretty simple and reasonable way to make the site a whole lot less attractive to some of the most obnoxious sort of spammers. I would hope nobody is adding links to their SO answers with the expectation of an SEO benefit.

bryanlarsen 14y ago
If there's a link in a stackoverflow answer of mine, it's there because there's really good information at the end of the link. That's the sort of information Google & Bing use to make their searches useful, it's the sort of information I want Google et al to know.
- drivebyacct2 14y ago
  Yes, I agree. StackOverflow is harming the quality of results in Google because they want people to filter through them for the information. Anyone that's used SO knows that answers can be misleading, incomplete or straight up misinformed. At the very least they're generally less comprehensive and less keyword-full than source material that would appear as links. I suppose it's selfish of me to want SO to share them and selfish of SO to mark them as nofollow and not allow the source articles to rise in ranking.
  - vitalique 14y ago
    Original topic is not explicitly about content, but speaking of SO's content: > StackOverflow is harming the quality of results in Google because they want people to filter through them for the information. Well, no, actually. StackOverflow may be shooting itself in the foot. SO cannot dictate the rules of the search game to Google, and Google adjusts its SERPs pretty quickly, especially nowadays, when user behavior gets to have bigger influence on SERP with every ranking algo update. Poor quality content leads to higher bounce rate and lower avg. time on site and visit depth, which will inevitably lead to lower ranking and less traffic for SO. This (and perhaps a thousand of other factors) works for every site, and even more so for heavy traffic content projects like StackOverflow. So don't be afraid, the system will adjust itself. Overall, I agree with your sentiment: I love SO, but nofollowing the Web looks kind of lame and selfish. After all, doesn't SO have its staff of moderators and admins to fight spammy comments?

jeremysmyth 14y ago

This topic is discussed quite frequently in the meta pages on StackOverflow. For example, http://meta.stackoverflow.com/questions/111279/remove-nofollow-on-links-deemed-reputable http://meta.stackoverflow.com/questions/111279/remove-nofoll... covers exactly this subject. SO gets lots of googlejuice, and because it's gameable, they appear to want to give some of that googlejuice only to "reputable" links (which they do for links in high-rep users' profiles). That's a really hard thing to judge, but the fact that they're thinking about it, talking about it, and soliciting feedback on it means that they're trying to do the right thing, even if the article's author doesn't think their "right thing" is good enough for him.

VMG 14y ago

The author doesn't address this response by Jeff Atwood: URGENT UPDATE We were seeing a significant drop in Google (organic) traffic for Server Fault after instituting this "follow links if enough upvotes post-edit or post-create" policy. We traced it back to what we currently think are a string of posts on Server Fault that got nofollow removed through "trust", but were being interpreted by Google as link farms or spammy pages. [....] http://meta.stackoverflow.com/a/51156 http://meta.stackoverflow.com/a/51156

bryanlarsen 14y ago
There's a follow-up to the URGENT UPDATE on SO: http://meta.stackoverflow.com/questions/111279/remove-nofollow-on-links-deemed-reputable http://meta.stackoverflow.com/questions/111279/remove-nofoll... So it appears that they do remove rel=nofollow from reputable links, although their threshold for reputable appears to be very high.
preinheimer 14y ago
How is that not the expected result? When the sites that informed various well regarded answers on the sites get their due, their search engine rank rises. With such a well ranked site now pointing towards them, they rise in the results, often higher than the SO answer referencing it. As a secondary issue, there's a relatively common black-hat-seo practice of buying a well regarded domain, and soaking that links for all they're worth to promote something. Their switch-flipping may have run afoul of systems designed to mitigate that.
kamjam 14y ago
The author posted the original article on 17th Dec 2010 and an update on 11th Dec 2011. So not sure how this made it to the front page of HN. Not surprised the author doesn't address the response.

cstrat 14y ago

Someone have a mirror of the article?

cstrat 14y ago
google cache seems ok: http://webcache.googleusercontent.com/search?q=cache:3sKnQOnbA0UJ:www.brianbondy.com/blog/id/104/stackoverflow-amongst-nofollow-web-abuse-sites+&cd=1&hl=en&ct=clnk&gl=au http://webcache.googleusercontent.com/search?q=cache:3sKnQOn...

kmontrose 14y ago

This was dealt with more than a year ago, when we did remove nofollow from trusted posts links' http://meta.stackoverflow.com/questions/111279/remove-nofollow-on-links-deemed-reputable http://meta.stackoverflow.com/questions/111279/remove-nofoll.... Would have been dealt with even earlier, but our experiments with it (http://meta.stackoverflow.com/a/69032/130213 http://meta.stackoverflow.com/a/69032/130213) lead us to believe Google was starting to treat us as a link farm. What we were seeing in our earlier attempt wasn't pages appearing lower in results (which could reasonably be expected in some cases as a result of the change; and we wouldn't care, 2nd on a search under an equally good resource is fine by us), but pages being removed entirely; almost universally pages that had newly un-nofollow-ed links to stores, or ad-laden sites (mostly legitimate posts, but some spam that stuck around for a bit before the community deleted it). So, "classified as link farm" seemed like the most likely problem. Google naturally won't tell you why your ranking drops, so it was (and remains) just an educated guess as to what happened. So we stopped, putting it on the "wish we could, but reality doesn't let us"-list until Google reached out to us (among thousands of others, I'm sure) to change our nofollow practices. Google didn't describe any changes in their algorithm, but it seems reasonable to me that there would have been some tweaks around nofollow to accompany a new policy; again just an educated guess. Basically, this is an old post complaining about a long since addressed concern that we had tried to address even earlier but ran into practical problems with. While the exact details of our algorithm are secret by necessity, I will say that we've had to consider posts individually to prevent spammers from kiting a single account up to post nofollow-less spam content. People still try, it's kind of astounding how much spammers try (I suspect SEO's opaqueness cuts both ways here), but it doesn't work (well, you can get one link in your profile; but there's less SEO juice to pass and you are hard-capped at one, no matter how long it takes someone to delete your account). Disclaimer: Stack Exchange employee, I was on all the relevant calls but has been a couple years so grain of salt and all that.

nikatwork 14y ago
I've seen this happen to other sites, something angers the Google algo and organic traffic falls off a cliff. History seems to bear out that SO correctly identified the source of the "problem", and SO is such a valuable resource that I'd rather it be Google-visible over passing pagerank. This is the problem when you have a single player dominating the search space with an opaque algo and a very limited appeals process.

bhanks 14y ago

How is this news? 1. This post is from 2011 2. Google ended the ability to link sculpt a long time ago. I would be interested in studies showing cases where link sculpting still worked.

kamjam 14y ago
Agree. The original post is actually from from Dec 2010 and an update on Dec 2011!

robomartin 14y ago

I really don't see SO as being the problem here. I happen to think that secret SEO/page rank algorithms are the culprit in many ways. Google is in the unique position as to be able to write a "how to behave" guide for the internet. Play by their rules and you are golden, start to violate them and you start to be "punished" in a gradual (but exponentially more severe). The whole time full feedback and data on what you are doing wrong is provided. As this approach evolves spammers would be less and less successful and might just have to resort to legitimate means of marketing.

JangoSteve 14y ago
Furthermore, the search engines could nip this kind of abuse in the bud, by ignoring the "nofollow" attribute of the links on a site if it determines that site is abusing it (e.g. if all or some high percentage of the links on the site have "nofollow").

pootch 14y ago

SO has serious content problems of which nofollow is not one of them. SO cant decide if it wants deep content or shallow content, and it has mostly ended up being shallow (and massive amounts of incorrect deep/shallow content), links outside of SO are a major PAIN, as SO is in the format business not the content business. If the "answer" is not in SOs format which sometimes barely ranks above commenting, then its a bad thing. If you have blogworthy content dont bother posting it on SO, its a waste of time and in the Q&A format of SO its also a nuisance.

netcan 14y ago

This nofollow stuff has always seemed very weird to me for several reasons. - Search Engines are supposed to be analyzing the web, figuring out whats important. They have the incentive to do this well. The websites they are analyzing do not. Websites have an incentive to nofollow everything. They might be worried about their own rankings, but why should they care about the sites they link to. Nofollow is safe and harmless. No-nofolow should get you in trouble. what does a site gain from not using nofollow links on everything? - To enforce nofollow rules, Google are supposedly disciplining sites by hurting their organic rankings. But a page with nofollow links appears exactly the same to a user. It's just as right an answer as it was before. Is Google lowering the quality of its results to police the web? - If Google are able to detect "link farms" and user generated content that should have been nofollow-ed, why don't they just treat those links as nofollow and ignore them. Use this detection to analyze the web, not police it. - Are they ignoring important information sources? A huge, information rich portion of the web is user generated. Eg wikipedia & stackoverflow. How can Google really be ignoring these links here as data sources? The links on a wikipedia page for example, are very informative. If some webpage is mentioned frequently in stackoverflow questions and answers, it's probably important and its probably a good answer to a lot of questions people are asking google.

doyouevenlift 14y ago

No one cares.

205guy 14y ago

Esprit d'escalier: "In my day (usenet), we didn't have links--and we liked it." (old SNL joke for those who don't get it). This conversation has 2 angles I find very interesting. The first is the whole question of quality of the content. Google is a huge AI trying to give you the same quality of information that you'd select if you looked at the same (millions of) sources. At first, I liked netcan's answer[1], which says essentially, nofollow them all or none, Google will sort out its own. But the more I think about it, the more context that sites can provide automatically, the better. If a site can say "this content provided by untrusted source" (because we're a community-driven site, and we can't police everything), that's a help to Google's Algorithm. Google is free to still follow the link and perhaps, using it's other knowledge of the target, assign a trustworthiness score to the nofollow quality on the original site. If a good community is providing good links, but the website owner still marks them nofollow, that might become a badge of honor in Google eyes (all-crawling robot tentacles?). As a side note, I love how everybody, from slimy SEO to internet ethicist, is trying to guess Google's secret formula (it used to be Coca-Cola that had a secret formula that was releveant). Google is essentially the unknowable proto-God of this new information universe. The other interesting topic is getting paid for content. My first reaction to the OP was "wah, wah, OP doesn't want to pay for his little corner of the internet." Let's face it, the internet doesn't run for free (hardware and electricity), and the content that attracts people doesn't write itself. StackOverflow provides a community for people to deliver their content into and receive socaial rewards. If SO can't do that because of some quirk of SEO, then it's likely it wouldn't be able to host the community anymore. The internet has still not figured out a way to pay for that other than advertizing (the forcing of unwanted content on readers). But advertizing only works if you can keep track of views and stay on top of the SEO game. I totally agree they're hypocrites for hoarding the follows (must follow to us be we nofollow to everyone), but how's that different from a corporate board's fiduciary duty to shareholders to maximize the profit of the company? I also think cheald[2] makes a very good point about why the dynamics must be towards nofollow. The question is then how involved or committed are you to the community, and how do you feel rewarded for the time you contribute to it. [1] http://news.ycombinator.com/item?id=4777260 http://news.ycombinator.com/item?id=4777260 [2] http://news.ycombinator.com/item?id=4774888 http://news.ycombinator.com/item?id=4774888