6 ms·

It's not a very difficult problem to solve. 95% of the content-farm spam comes from a few domains. In the same way that spam-blacklists have proved to be the

It's not a very difficult problem to solve. 95% of the content-farm spam comes from a few domains. In the same way that spam-blacklists have proved to be the most-effective way to combat e-mail spam, Google just needs to decide to shut these content-farms out. They don't need to do anything sophisticated like tweak their algorithm... just shut them out. The fact that it hasn't been done yet suggests to me that Google doesn't want to.

jules 16y ago
Do these content farms run Google ads?
srean 16y ago
Perhaps because of legal issues, particularly when Google is being sued for tweaking its ranking algorithm to boost their own pages higher. I dont understand the legalities of that lawsuit though. A ranking is an opinion. Its not that its some fundamental physical constant or a formally well defined quantity. Can I be sued for my subjective judgment ? OT: Will appreciate if you let me know why you down voted.
- roadnottaken 16y ago
  I don't understand those lawsuits either. Isn't Google a private company? Aren't they entitled to do what they want with their product? I see no reason why their results must be "fair", legally.
  - barrkel 16y ago
    Well, if they're actually a monopoly, then it does change what they can do, legally.
    - bad_user 16y ago
      IANAL, but proving that a company has a monopoly and used that monopoly in an abusive way is very hard. Consider that when discussing whether a company has a monopoly on a market, alternatives and cost of switching also comes up (besides market share); so it will be even harder than in the case of Microsoft. Then even if Google is discovered to have a monopoly, shutting down a website ... can be argued that it was in the best interests of its users and faithful to the product's original mission: i.e. nobody can sue Microsoft for improving Windows / not bundling third-party software (they got sued for extending Windows with new functionality that destroyed competition in an existing market).
      - ZachPruckowski 16y ago
        can be argued that it was in the best interests of its users and faithful to the product's original mission If you're willing to pay the tens of millions of dollars and take the PR hit that is a government anti-trust lawsuit, you may eventually have the privilege of making that argument before a federal judge, who probably isn't technology literate enough to comprehend it. Let's face facts here, this conversation goes over the heads of 80% of internet users. The judge is going to see "the federal government accuses them of restraining this site's trade by removing them from the search index, and Google admits to it" and then that's the ballgame. Or maybe the government never gets around to suing Google. Or maybe they get a judge who knows this sort of stuff already. But that's still a scenario keeping Google's Legal Dept. up at night.
        bad_user 16y ago
        Yeah, but that would be like suing Microsoft for making Windows more secure because that can "restrain the trade of companies producing Antiviruses". That would be a pretty stupid argument, even in the face of a non-technical jury, wouldn't it? Of course, the negative image would hurt, but lets be honest, the suit against Microsoft accomplished basically nothing it couldn't handle, while waisting taxpayer's money. Are these antitrust suits getting started so easily?
        jarek 16y ago
        Or like suing Microsoft for making the computers that ship with its OS more useful by including Office cheaply...
      - eftpotrm 16y ago
        I would be surprised to see them suffering for optimising the algorithm to try and trace the original source of content though. With the sort of sites we're talking about, by definition that original source will be more up-to-date content which the user could reasonably expect to be presented preferentially. At which point - you're welcome to make your business from rehosting public content that originates elsewhere, but if you expect us to primarily direct people to your copy rather than the original then I think you're onto a loser. Let's put it another way. I could conceivably build a valuable service by taking content from StackOverflow and Wikipedia (or wherever) and automatically linking the two to enable people to get some more context around some questions, maybe reformatting pages to enable side-by-side content or something similar. It wouldn't be a trivial service but it wouldn't be impossible in the least and it could plausibly add value. As such it wouldn't be unreasonable to preferentially direct users in some cases to that source rather than the original, as algorithmically optimised content - the preferential ranking would be a result of the value of the linking algorithm. Without this though, by what measure am I conceivably superior to the original source by having an out-of-date copy with fewer legitimate inbound links and more irrelevant content (adverts)? Being a monopoly isn't illegal. Using your monopoly power in a way that might harm another company isn't illegal. What's illegal is doing that unreasonably and capriciously, as Microsoft were with Netscape in that trial, or DR and Lotus in previous trials.
    - sethg 16y ago
      No law, as far as I am aware, forbids a monopoly from producing a crappy product. Antitrust law comes into play when a competitor offers a less-crappy product and the monopoly tries to drive the competitor out of business.
      - rbanffy 16y ago
        No, but a company that has a monopoly cannot manipulate a market by making a product selectively crappy, like a search engine that fails to find its competitors or a program that fails to run on a competing, compatible OS (early win3 on DR-DOS).
        sethg 16y ago
        Yes, if you make your product selectively crappy in order to undermine competition, it’s an antitrust problem. Microsoft making Windows not run on DR-DOS is an antitrust problem. Microsoft making Windows fail at multiuser security is not an antitrust problem.
        rbanffy 16y ago
        > Microsoft making Windows fail at multiuser security is not an antitrust problem. No. That was incompetence.
  - natep 16y ago
    It's a publicly traded company, but yes, "private" if you mean "not controlled by a government"
duffbeer703 16y ago
Or can't. It's hard to make an argument that you are not a monopoly when you put Demand Media out of business in 30 days by blacklisting them.
jwhite 16y ago
It reminds me of the reddit IAMA a while back by an affiliate marketer [1]. There were lots of responses by people claiming he was the scum of the earth, poisoning the web, not adding value, but he just calmly stated his position that affiliate marketers are a significant source of income for Google, and "Google == value", therefore affiliate marketing has value. Same might be said for content farms, so I reckon your last sentence is on the money. [1] http://www.reddit.com/r/IAmA/comments/azcni/i_made_62232296_in_2009_from_affiliate_marketing/ http://www.reddit.com/r/IAmA/comments/azcni/i_made_62232296_...
- kmfrk 16y ago
  Hacker mentality is largely gaming systems, so Google should indeed change the rules.
  - jwhite 16y ago
    I wish they would. I find content farms irksome.
cletus 16y ago
Two points: 1. What makes you claim blacklists are the "most-effective" form of filtering? My understanding is that other techniques like Bayesian filtering are. Background: http://www.paulgraham.com/falsepositives.html http://www.paulgraham.com/falsepositives.html 2. Saying that it only comes from a few sources is dangerous thinking. This type of thinking was used by politicians to get a ban on adult services from Craigslist ("get rid of it there and it goes away!").
- fanf2 16y ago
  A good email blacklist (such as the Spamhaus Zen list) will detect 90% of spam with no false positives and negligible load. There is a lot of evidence that there are not many botnets and each one has little diversity of control. This observation does not generalize to other kinds of spam such as 419s.
sethg 16y ago
But if Google shuts those domains out, then the content farmers can just pick up new ones. Furthermore, Google would need to have some sort of process for determining which domains are content-farm spam before shutting them out. As with spam, the bad guys can be far more agile than the good guys. The point of having algorithms is that once you have a good algorithm, your search-quality staff doesn’t have to scale linearly with the number of pages on the Web. (Disclaimer: I work in the search division of Nokia, i.e., I work for one of Google’s competitors.)
- ekanes 16y ago
  Still, a power law applies and you can get huge gains by punishing the biggest offenders and moving down from there. And yes, spammers can always restart, but the sandbox helps with that.
- roadnottaken 16y ago
  Google has 23,000 employees and ~$10B in yearly operating income. I'm pretty sure they could stay ahead of the bad guys if they wanted to. EDIT: Additionally, it is hard to game pagerank rapidly because the rest of the web needs to link to your site. So, even if you switch your knock-off wikipedia site to a new domain, it would take weeks/months to rise through the rankings. I'm pretty sure it would only take a few employees (at most) to stay ahead of these huge spam sites.
  - rokhayakebe 16y ago
    it is hard to game pagerank rapidly because the rest of the web needs to link to your site Not really. Some people own different websites, purchased through different accounts, hosted on different servers. They, then, link between those sites.
  - gregholmberg 16y ago
    Google has 23,000 employees ... . I'm pretty sure they could stay ahead of the bad guys if they wanted to. In the past there were people within Google who had the skill, knowledge, organizational connections and authority to quickly and gracefully reduce the problem of link spam without harming the bottom line too much. I think we all loved their early work, and our love helped propel their little company to multibillions-per-year. I don't think that kind of unique early experience and subsequent problem-solving effectiveness is snap-in interchangeable. Some of those "rock stars" work at Facebook now. Google might not be capable of fixing the problem gracefully without them.

jules 16y ago

Do these content farms run Google ads?

srean 16y ago

Perhaps because of legal issues, particularly when Google is being sued for tweaking its ranking algorithm to boost their own pages higher. I dont understand the legalities of that lawsuit though. A ranking is an opinion. Its not that its some fundamental physical constant or a formally well defined quantity. Can I be sued for my subjective judgment ? OT: Will appreciate if you let me know why you down voted.

roadnottaken 16y ago
I don't understand those lawsuits either. Isn't Google a private company? Aren't they entitled to do what they want with their product? I see no reason why their results must be "fair", legally.
- barrkel 16y ago
  Well, if they're actually a monopoly, then it does change what they can do, legally.
  - bad_user 16y ago
    IANAL, but proving that a company has a monopoly and used that monopoly in an abusive way is very hard. Consider that when discussing whether a company has a monopoly on a market, alternatives and cost of switching also comes up (besides market share); so it will be even harder than in the case of Microsoft. Then even if Google is discovered to have a monopoly, shutting down a website ... can be argued that it was in the best interests of its users and faithful to the product's original mission: i.e. nobody can sue Microsoft for improving Windows / not bundling third-party software (they got sued for extending Windows with new functionality that destroyed competition in an existing market).
    - ZachPruckowski 16y ago
      can be argued that it was in the best interests of its users and faithful to the product's original mission If you're willing to pay the tens of millions of dollars and take the PR hit that is a government anti-trust lawsuit, you may eventually have the privilege of making that argument before a federal judge, who probably isn't technology literate enough to comprehend it. Let's face facts here, this conversation goes over the heads of 80% of internet users. The judge is going to see "the federal government accuses them of restraining this site's trade by removing them from the search index, and Google admits to it" and then that's the ballgame. Or maybe the government never gets around to suing Google. Or maybe they get a judge who knows this sort of stuff already. But that's still a scenario keeping Google's Legal Dept. up at night.
      - bad_user 16y ago
        Yeah, but that would be like suing Microsoft for making Windows more secure because that can "restrain the trade of companies producing Antiviruses". That would be a pretty stupid argument, even in the face of a non-technical jury, wouldn't it? Of course, the negative image would hurt, but lets be honest, the suit against Microsoft accomplished basically nothing it couldn't handle, while waisting taxpayer's money. Are these antitrust suits getting started so easily?
        jarek 16y ago
        Or like suing Microsoft for making the computers that ship with its OS more useful by including Office cheaply...
    - eftpotrm 16y ago
      I would be surprised to see them suffering for optimising the algorithm to try and trace the original source of content though. With the sort of sites we're talking about, by definition that original source will be more up-to-date content which the user could reasonably expect to be presented preferentially. At which point - you're welcome to make your business from rehosting public content that originates elsewhere, but if you expect us to primarily direct people to your copy rather than the original then I think you're onto a loser. Let's put it another way. I could conceivably build a valuable service by taking content from StackOverflow and Wikipedia (or wherever) and automatically linking the two to enable people to get some more context around some questions, maybe reformatting pages to enable side-by-side content or something similar. It wouldn't be a trivial service but it wouldn't be impossible in the least and it could plausibly add value. As such it wouldn't be unreasonable to preferentially direct users in some cases to that source rather than the original, as algorithmically optimised content - the preferential ranking would be a result of the value of the linking algorithm. Without this though, by what measure am I conceivably superior to the original source by having an out-of-date copy with fewer legitimate inbound links and more irrelevant content (adverts)? Being a monopoly isn't illegal. Using your monopoly power in a way that might harm another company isn't illegal. What's illegal is doing that unreasonably and capriciously, as Microsoft were with Netscape in that trial, or DR and Lotus in previous trials.
  - sethg 16y ago
    No law, as far as I am aware, forbids a monopoly from producing a crappy product. Antitrust law comes into play when a competitor offers a less-crappy product and the monopoly tries to drive the competitor out of business.
    - rbanffy 16y ago
      No, but a company that has a monopoly cannot manipulate a market by making a product selectively crappy, like a search engine that fails to find its competitors or a program that fails to run on a competing, compatible OS (early win3 on DR-DOS).
      - sethg 16y ago
        Yes, if you make your product selectively crappy in order to undermine competition, it’s an antitrust problem. Microsoft making Windows not run on DR-DOS is an antitrust problem. Microsoft making Windows fail at multiuser security is not an antitrust problem.
        rbanffy 16y ago
        > Microsoft making Windows fail at multiuser security is not an antitrust problem. No. That was incompetence.
- natep 16y ago
  It's a publicly traded company, but yes, "private" if you mean "not controlled by a government"

duffbeer703 16y ago

Or can't. It's hard to make an argument that you are not a monopoly when you put Demand Media out of business in 30 days by blacklisting them.

jwhite 16y ago

It reminds me of the reddit IAMA a while back by an affiliate marketer [1]. There were lots of responses by people claiming he was the scum of the earth, poisoning the web, not adding value, but he just calmly stated his position that affiliate marketers are a significant source of income for Google, and "Google == value", therefore affiliate marketing has value. Same might be said for content farms, so I reckon your last sentence is on the money. [1] http://www.reddit.com/r/IAmA/comments/azcni/i_made_62232296_in_2009_from_affiliate_marketing/ http://www.reddit.com/r/IAmA/comments/azcni/i_made_62232296_...

kmfrk 16y ago
Hacker mentality is largely gaming systems, so Google should indeed change the rules.
- jwhite 16y ago
  I wish they would. I find content farms irksome.

cletus 16y ago

Two points: 1. What makes you claim blacklists are the "most-effective" form of filtering? My understanding is that other techniques like Bayesian filtering are. Background: http://www.paulgraham.com/falsepositives.html http://www.paulgraham.com/falsepositives.html 2. Saying that it only comes from a few sources is dangerous thinking. This type of thinking was used by politicians to get a ban on adult services from Craigslist ("get rid of it there and it goes away!").

fanf2 16y ago
A good email blacklist (such as the Spamhaus Zen list) will detect 90% of spam with no false positives and negligible load. There is a lot of evidence that there are not many botnets and each one has little diversity of control. This observation does not generalize to other kinds of spam such as 419s.

sethg 16y ago

But if Google shuts those domains out, then the content farmers can just pick up new ones. Furthermore, Google would need to have some sort of process for determining which domains are content-farm spam before shutting them out. As with spam, the bad guys can be far more agile than the good guys. The point of having algorithms is that once you have a good algorithm, your search-quality staff doesn’t have to scale linearly with the number of pages on the Web. (Disclaimer: I work in the search division of Nokia, i.e., I work for one of Google’s competitors.)

ekanes 16y ago
Still, a power law applies and you can get huge gains by punishing the biggest offenders and moving down from there. And yes, spammers can always restart, but the sandbox helps with that.
roadnottaken 16y ago
Google has 23,000 employees and ~$10B in yearly operating income. I'm pretty sure they could stay ahead of the bad guys if they wanted to. EDIT: Additionally, it is hard to game pagerank rapidly because the rest of the web needs to link to your site. So, even if you switch your knock-off wikipedia site to a new domain, it would take weeks/months to rise through the rankings. I'm pretty sure it would only take a few employees (at most) to stay ahead of these huge spam sites.
- rokhayakebe 16y ago
  it is hard to game pagerank rapidly because the rest of the web needs to link to your site Not really. Some people own different websites, purchased through different accounts, hosted on different servers. They, then, link between those sites.
- gregholmberg 16y ago
  Google has 23,000 employees ... . I'm pretty sure they could stay ahead of the bad guys if they wanted to. In the past there were people within Google who had the skill, knowledge, organizational connections and authority to quickly and gracefully reduce the problem of link spam without harming the bottom line too much. I think we all loved their early work, and our love helped propel their little company to multibillions-per-year. I don't think that kind of unique early experience and subsequent problem-solving effectiveness is snap-in interchangeable. Some of those "rock stars" work at Facebook now. Google might not be capable of fixing the problem gracefully without them.