3 ms·

Here's the problem - how do you 1. define SEO spam, 2. detect SEO spam. It's not as simple as saying "deindex anything that has optimized title tags or lots of

by phpnode 16y ago

Here's the problem - how do you 1. define SEO spam, 2. detect SEO spam. It's not as simple as saying "deindex anything that has optimized title tags or lots of links or shows adsense" because plenty of useful websites optimise for search engines, show ads and have lots of links. What you'd end up with is the dregs of the web, because although you'd reduce the number of spam sites in the results, you'd likely also reduce the number of useful sites in the results because you're specifically targeting and removing the sites that optimise for search engines, which is 99% of the most successful sites on the web today. SEO is not an inherently bad thing, it's valueless, spammy content that's bad. The real challenge is in determining whether a page is useful, original and relevant to the search query, and that's where Google et al are falling down at the moment.

teyc 16y ago
No, Google is being gamed. If you follow the money, e.g. AdSense, Affiliate Links, you will eventually disincentivize SEO spam.
- phpnode 16y ago
  Right, but you'll also disincentivize legitimate websites that happen to make their money through such ads or affiliate links. Where do you draw the line?
  - stcredzero 16y ago
    The solution is simple here: have an alternative search engine that makes their money a different way. Even though it would just be a niche, I bet there's a pretty healthy income to be had for a highly SEO-spam resistant paid search engine. (Supplemented with Google-style search ads.) A small minority would use such a search engine, so the impact on legitimate sites would be minimal.
    - phpnode 16y ago
      If your new search engine gets enough traffic to be worth it, spammers will just target your search engine too, and you'll have exactly the same problem Google has now, but presumably with far fewer engineering resources to throw at the problem.
      - stcredzero 16y ago
        If your new search engine gets enough traffic to be worth it, spammers will just target your search engine too I think you're dead wrong here. I only need to catch a sliver of Google/Bing traffic for such a search engine to be worth it. Let's say I get a 1% share of traffic -- which I would characterize as wildly successful -- why would it be worth the while of SEO spammers? How many resources can you throw at something worth 1% of your income?
        phpnode 16y ago
        Because spam is cheap and easy, far easier to produce than to fight, you can spam in your sleep with automated tools. Let's take your 1% example, that's millions of searches a day, well worth a spammer's time and effort. Some people will spam your search engine just because you say they can't, and that kills your unique selling point straight away.
        stcredzero 16y ago
        Because spam is cheap and easy, far easier to produce than to fight, you can spam in your sleep with automated tools. What you are saying is that that someone could algorithmically determine the profitable targets of spam. This opens the door to an economic hack of spamming. Simply run the same algorithms yourself, then if the spam-profit potential of a particular search is above a certain threshold, simply present a "hammy" search based on well-known authoritative sites. This scheme would work, precisely because spam is cheap and easy. Some people will spam your search engine just because you say they can't, and that kills your unique selling point straight away. This only works against the scheme I just described if you can spam my method for determining "well-known authoritative sites." If you curate that, then you can't. What I'm advocating is actually switching between curated and algorithmic search depending on spam-economics. This means that spamming such a site might end up being equivalent to "internally spamming Wikipedia and StackOverflow." I don't think that's going to be viable to SEO spammers for going after an extra 1% income.
  - teyc 16y ago
    If Google serves AdSense, it already knows how long you as a reader spends on site before clicking through an ad. Presumably, if statistically speaking, visitors have high probability of hitting the back button or clicking through immediately, then it is a red flag that the content wasn't useful.
stcredzero 16y ago
It's not as simple as saying "deindex anything that has optimized title tags or lots of links or shows adsense" No, it's not. But undoubtedly, SEO spammers are leaving behind the same kind of clues that email spammers do. Using techniques like Bayesian classifiers should be able to classify such pages. Just detecting all the pages that are slightly edited/SEO-optimized stackoverflow answers would be a good start for me. you'd likely also reduce the number of useful sites in the results because you're specifically targeting and removing the sites that optimise for search engines, which is 99% of the most successful sites on the web today. I'd be putting useful sites in the "useful" corpus. SEO is not an inherently bad thing, it's valueless, spammy content that's bad. Agreed, just as email marketing is not an inherently bad thing, just spammy emails. http://xkcd.com/810/ http://xkcd.com/810/
- phpnode 16y ago
  There are billions and billions of pages in the index, let's say they produce a page classifier that can correctly identify spam an unrealistic 99% of the time. They'd still misidentify tens of millions of pages. Regarding your second point, how do you determine usefulness in a way that can't be gamed? My point is, this is a very, very, very hard problem to solve at their scale, a lot of posters on here are trivialising it.
  - stcredzero 16y ago
    99% classifier? So I get to have a search that's like Google/Bing for all the non-spammy searches, and it only gets 1/100th the spam for the spammy searches? I think I could sell a proposition like that!

by phpnode 16y ago

teyc 16y ago

No, Google is being gamed. If you follow the money, e.g. AdSense, Affiliate Links, you will eventually disincentivize SEO spam.

phpnode 16y ago
Right, but you'll also disincentivize legitimate websites that happen to make their money through such ads or affiliate links. Where do you draw the line?
- stcredzero 16y ago
  The solution is simple here: have an alternative search engine that makes their money a different way. Even though it would just be a niche, I bet there's a pretty healthy income to be had for a highly SEO-spam resistant paid search engine. (Supplemented with Google-style search ads.) A small minority would use such a search engine, so the impact on legitimate sites would be minimal.
  - phpnode 16y ago
    If your new search engine gets enough traffic to be worth it, spammers will just target your search engine too, and you'll have exactly the same problem Google has now, but presumably with far fewer engineering resources to throw at the problem.
    - stcredzero 16y ago
      If your new search engine gets enough traffic to be worth it, spammers will just target your search engine too I think you're dead wrong here. I only need to catch a sliver of Google/Bing traffic for such a search engine to be worth it. Let's say I get a 1% share of traffic -- which I would characterize as wildly successful -- why would it be worth the while of SEO spammers? How many resources can you throw at something worth 1% of your income?
      - phpnode 16y ago
        Because spam is cheap and easy, far easier to produce than to fight, you can spam in your sleep with automated tools. Let's take your 1% example, that's millions of searches a day, well worth a spammer's time and effort. Some people will spam your search engine just because you say they can't, and that kills your unique selling point straight away.
        stcredzero 16y ago
        Because spam is cheap and easy, far easier to produce than to fight, you can spam in your sleep with automated tools. What you are saying is that that someone could algorithmically determine the profitable targets of spam. This opens the door to an economic hack of spamming. Simply run the same algorithms yourself, then if the spam-profit potential of a particular search is above a certain threshold, simply present a "hammy" search based on well-known authoritative sites. This scheme would work, precisely because spam is cheap and easy. Some people will spam your search engine just because you say they can't, and that kills your unique selling point straight away. This only works against the scheme I just described if you can spam my method for determining "well-known authoritative sites." If you curate that, then you can't. What I'm advocating is actually switching between curated and algorithmic search depending on spam-economics. This means that spamming such a site might end up being equivalent to "internally spamming Wikipedia and StackOverflow." I don't think that's going to be viable to SEO spammers for going after an extra 1% income.
- teyc 16y ago
  If Google serves AdSense, it already knows how long you as a reader spends on site before clicking through an ad. Presumably, if statistically speaking, visitors have high probability of hitting the back button or clicking through immediately, then it is a red flag that the content wasn't useful.

stcredzero 16y ago

It's not as simple as saying "deindex anything that has optimized title tags or lots of links or shows adsense" No, it's not. But undoubtedly, SEO spammers are leaving behind the same kind of clues that email spammers do. Using techniques like Bayesian classifiers should be able to classify such pages. Just detecting all the pages that are slightly edited/SEO-optimized stackoverflow answers would be a good start for me. you'd likely also reduce the number of useful sites in the results because you're specifically targeting and removing the sites that optimise for search engines, which is 99% of the most successful sites on the web today. I'd be putting useful sites in the "useful" corpus. SEO is not an inherently bad thing, it's valueless, spammy content that's bad. Agreed, just as email marketing is not an inherently bad thing, just spammy emails. http://xkcd.com/810/ http://xkcd.com/810/

phpnode 16y ago
There are billions and billions of pages in the index, let's say they produce a page classifier that can correctly identify spam an unrealistic 99% of the time. They'd still misidentify tens of millions of pages. Regarding your second point, how do you determine usefulness in a way that can't be gamed? My point is, this is a very, very, very hard problem to solve at their scale, a lot of posters on here are trivialising it.
- stcredzero 16y ago
  99% classifier? So I get to have a search that's like Google/Bing for all the non-spammy searches, and it only gets 1/100th the spam for the spammy searches? I think I could sell a proposition like that!