9 ms·

Google Destroyed the Web

38 points by dkasper 17y ago

I think the last point is the most interesting. Do we need a search engine that punishes sites with ads to get rid of spam?

jacquesm 17y ago
Google destroyed the web much more by using pagerank than they ever did by adsense. The simple cure for MFA sites would be to simply ban those pages that use google adsense from search engine results pages. 10 minute fix. Tops. But it will also seriously impact googles revenues and that is why it isn't going to happen. A really good argument against having too many of these services in the hands of the same company. The bigger problem, and one that is much harder to fix is that by using the structure of the web to measure page popularity google has put a premium on manipulating the structure of the web for profit. The damage that has done is a lot bigger than MFA sites, those only affect search engine results. It is a logical consequence of the ranking method used by a search engine that the damage is limited to that which is being measured. Altavista led to keyword spam which was limited to the pages of the sites owned by those trying to gain as much traffic as possible from it, Google has led to link spam for exactly the same reason.
- alain94040 17y ago
  The simple cure would be to simply ban those pages that use google adsense This is actually an amazingly good idea! It should be possible to build such a product on top of Google search. And you have a competitive advantage against Google: they will never ban sites that use AdSense, it would kill their $10B market. So you are safe.
  - jacquesm 17y ago
    I somehow doubt google would be pleased if you did that on Google search. It also leaves you with the subtle problem of where you're going to find a way to monetize, adsense would be 'out'. edit: another way google could fix this is by no longer counting visitors that they send to sites when those visitors click on adsense ads. In other words your adsense income can not come from selling google traffic back to google. That would likely cure 90% of the problem, since these sites have 0 autonomous traffic.
    - alain94040 17y ago
      where you're going to find a way to monetize You could put your own ads on your search results. The point is that people would prefer to search from your site, because your results would be even better (more relevant) than Google's. People wouldn't go to your site because they can't stand ads, but because they want to find links that are not filled with ads.
- gojomo 17y ago
  I have long wanted an 'advanced search operator' that excludes sites running AdSense ads. Similar to the 'site:' or 'link:' operators, we could have 'ads-from:'. So when I want sites without AdSense, I could add this to my query: -ads-from:google.com (I'm not holding my breath, though.)
  - jacquesm 17y ago
    If Microsoft wants to have some real fun with bing they add that option. I might even switch :)
  - prawn 17y ago
    I think there's potential for localised search with options such as: - lists prices online - is just brochureware - has online ordering - displays their full catalogue - etc Say I'm trying to get an idea of cost for tapware, I have to browse through x plumbing sites that are just brochureware, or have an only catalogue without pricing, etc. Similarly for travel, checkboxes for "official site", review site, reseller, etc. And another PITA is product review pages with SFA reviews.
- netcan 17y ago
  If having adsense ads reduced your search traffic, sites would just use some other ad service. Would you want to remove all sites using advertising of any kind? "Google bans advertising on all sites other than Google."
  - jacquesm 17y ago
    I think that is mostly a matter of degree. You could conceivably make it switchable so that you only cut out certain advertising networks. MFA seems to be by far the larger part of the problem, possibly other ad networks would see an increase but nobody has the advertiser volume required for that other than adsense. Of course the final outcome of something like that remains to be seen but I'm fairly sure it would be a net positive. And they could always default it to 'off'.
    - stcredzero 17y ago
      Google could develop some kind of metric for "too many ads" on a page. Just getting the "O" in SEO to mean some sort of moderation would be a kind fo victory.
      - netcan 17y ago
        I wouldn't be surprised if they do, or at least have considered it. The problem is that would be undermining other businesses with advertising based business models, their direct & indirect competitors. Google don't want to send you to made-for-adsense sites. These account for a tiny portion of their income while degrading both their search results & their advertising offering. Google' two primary concerns.
m0th87 17y ago
Has the author forgotten how impossible it was to navigate the WWW prior to Google? They essentially fixed search relative to the competitive solutions at the time (Lycos, AltaVista...shudder).
- dkasper 17y ago
  Perhaps he's thinking things seem to be regressing back to those days.
- dangrossman 17y ago
  It's not Google search that's the problem, it's Google AdSense.
  - jacquesm 17y ago
    Almost. It's the interaction between the two, google can't cut off the MFA sites from their serps without hurting their bottom line.
- lmoorman 17y ago
  That's like saying do you remember how hard it was to use a PC before Windows came around? The problems solved yesterday do not give you a pass today.
  - Semiapies 17y ago
    Yes, but anyone saying, "Windows/OS X/Linux/your OS here destroyed the PC" is more blatantly trolling than the linked post.
  - mattmcknight 17y ago
    It's actually not like saying that. He said Windows destroyed PCs.
- GHFigs 17y ago
  ...and now it's broken again, for different reasons. These things happen in cycles.
Perceval 17y ago
It might interesting to try and design a reverse-search engine, a kind of internet spam filter. Instead of being designed to promote relevant content, it would be designed to explicitly not return crap content. Rather than trying to improve Google's resiliency against noise, it might be more fruitful to turn the problem around.
- GHFigs 17y ago
  The answer is people. People you trust to not provide you with crap. Sometimes that's friends, sometimes that's family, sometimes that's strangers. It's never social media webcocks who want to "friend you", never bloggers offering you "n Ways to X your Y" for their $5/mo. in AdSense earnings, and only rarely the nebulous "crowd".
  - ramchip 17y ago
    Not so sure. When I'm looking for information on, say, WWII history, I can't ask someone or go on a trusted website like when I'm looking for information on a field I'm familiar with. I don't have any acquaintance or favourite websites already, which is why I'm using a search engine. The large amount of [quiz,farm,friend of the day] spam on my facebook, and the various hoax chain letters my not-too-web-savvy aunt sends us sometimes, also make me a little jaded on the value of 'social $whatever' for information gathering.
    - GHFigs 17y ago
      WWII history, I can't ask someone or go on a trusted website like when I'm looking for information on a field I'm familiar with. WWII history isn't such a hot topic for crapflooding, either. Search is still useful there. You're also still applying trust heuristics based on the results--"battle of the coral sea" returns results from Wikipedia, history.navy.mil, worldwar2history.info, history.sandiego.edu, etc. You probably have some idea about the content of each. Ultimately your decisions about what to click come down to your perception of trust in their authors' ability to provide you the information you desire. I'm just guessing, but I suspect most people here are familiar with the phenomenon of doing a Google search only to end up clicking on the Wikipedia result that invariably makes it into the top few results. This is a sign that Google's PageRank still aligns reasonably well with our own estimation of trust for those search terms. As for social whatevers: that's why I said "People you trust to not provide you with crap." You can't trust most people to do that. Least of all when given a megaphone and/or a financial incentive.
pmarsh 17y ago
If you want less page spam in your results write better queries into the search engine. There are still plenty of good results and it's fairly easy to ignore these SPAM pages just like you ignore banner ads. If the author said that Google's gotten worse then there might be reason to worry. But they're constantly improving I wouldn't worry about them getting beat at this game since they're the ones who control the rules.
stcredzero 17y ago
What he wants isn't a search engine that punishes ads. What he wants is really a search engine that punishes bad taste. (Specifically, too many ads greedily placed and greedily optimized text poorly written.) I think that such a search engine has tremendous profit potential. Would it be possible to detect redundancy? Is it possible to detect poor design?
- 10ren 17y ago
  Wasn't he was focusing mainly on the result not answering his question, rather than taste (though both are factors). At first I was thinking that what you seek amounts to strong AI, but then I remembered that there are measures for objectively assessing how easy text is to understand (even Word has some built-in); though, ironically, text that is understandable by a lower age is usually easier to understand (i.e. better) than that which is so convoluted that it requires multiple PhDs to parse. Unfortunately, Google is not finding these metrics for me, because I don't know what they're called... ah, here we go: Passive Sentences Test, Flesch Reading Ease and Flesch-Kincaid Grade Level.
  - stcredzero 17y ago
    I bet Google could do some simple metrics based on usefulness ratings by users (vote arrows) and number of ads. I bet they could also develop metrics to measure the same ratings against "degree of search engine optimization."
vaksel 17y ago
so how exactly does this guy propose content sites get paid if they can't use advertising to support themselves? Are they supposed to charge for access to read things that are readily available online?
- dangrossman 17y ago
  Real content sites aren't the problem, it's the sites that consist of nothing but thousands of pages of filler text and ads. A lot of them take articles off "free to reprint" article sites, and run them through programs that rearrange sentences and swap words with synonyms, to create "unique" content. Once Google finds the site, they start listing the pages, unable to differentiate between this garbage and real content.
  - vaksel 17y ago
    except Google has a strict policy against sites like that, and smacks them down the second they find them.
ThinkWriteMute 17y ago
There’s an opportunity here to create a web search engine that punishes results littered with ads. Google can’t do it – they live off those ads. This is absolute bullshit and it looks like a majority of replies here are eating it up. Google actively de-lists pages that aren't content intensive, and I mean serious legible content. Research some SEO IM work before you try and suggest that Google does nothing to remove Ad-sense filled pages. It's this kind of blogging that really muddies the waters.
- ThinkWriteMute 17y ago
  Looks like he denied my post directly on the blog. Real big of him. False alarm, just slow to approve. In parting I'd like to say this: wake up, Ad-riddled sites get tossed up in miliseconds (I know, I've written an engine to do so). You can't possibly make a search engine that can de-list fast enough. SEO IMers will fucking break you and that's just the way it's going to be unless someone (God forbid) regulates who can make websites and how fast. post script, there are millions, if not hundreds of millions, of legit websites that use the easy and unobtrusive Ad-sense ads to pay for (sometimes) very expensive hosting. You want to talk about destroying the internet? You're describing the obliteration of (almost) any non-corporate website.
johnl 17y ago
I've noticed the same problem, on some searches the entire first page is pure junk. Bet 80% of internet users don't know the difference. Maybe a power users mode that drops those from the search results would work.
bobbyi 17y ago
Today I searched for the answer to a question. The top hit was a useful article written by a subject-matter expert. (Good job, Google.) Not really destroyed then.

38 points by dkasper 17y ago

I think the last point is the most interesting. Do we need a search engine that punishes sites with ads to get rid of spam?

jacquesm 17y ago

Google destroyed the web much more by using pagerank than they ever did by adsense. The simple cure for MFA sites would be to simply ban those pages that use google adsense from search engine results pages. 10 minute fix. Tops. But it will also seriously impact googles revenues and that is why it isn't going to happen. A really good argument against having too many of these services in the hands of the same company. The bigger problem, and one that is much harder to fix is that by using the structure of the web to measure page popularity google has put a premium on manipulating the structure of the web for profit. The damage that has done is a lot bigger than MFA sites, those only affect search engine results. It is a logical consequence of the ranking method used by a search engine that the damage is limited to that which is being measured. Altavista led to keyword spam which was limited to the pages of the sites owned by those trying to gain as much traffic as possible from it, Google has led to link spam for exactly the same reason.

alain94040 17y ago
The simple cure would be to simply ban those pages that use google adsense This is actually an amazingly good idea! It should be possible to build such a product on top of Google search. And you have a competitive advantage against Google: they will never ban sites that use AdSense, it would kill their $10B market. So you are safe.
- jacquesm 17y ago
  I somehow doubt google would be pleased if you did that on Google search. It also leaves you with the subtle problem of where you're going to find a way to monetize, adsense would be 'out'. edit: another way google could fix this is by no longer counting visitors that they send to sites when those visitors click on adsense ads. In other words your adsense income can not come from selling google traffic back to google. That would likely cure 90% of the problem, since these sites have 0 autonomous traffic.
  - alain94040 17y ago
    where you're going to find a way to monetize You could put your own ads on your search results. The point is that people would prefer to search from your site, because your results would be even better (more relevant) than Google's. People wouldn't go to your site because they can't stand ads, but because they want to find links that are not filled with ads.
gojomo 17y ago
I have long wanted an 'advanced search operator' that excludes sites running AdSense ads. Similar to the 'site:' or 'link:' operators, we could have 'ads-from:'. So when I want sites without AdSense, I could add this to my query: -ads-from:google.com (I'm not holding my breath, though.)
- jacquesm 17y ago
  If Microsoft wants to have some real fun with bing they add that option. I might even switch :)
- prawn 17y ago
  I think there's potential for localised search with options such as: - lists prices online - is just brochureware - has online ordering - displays their full catalogue - etc Say I'm trying to get an idea of cost for tapware, I have to browse through x plumbing sites that are just brochureware, or have an only catalogue without pricing, etc. Similarly for travel, checkboxes for "official site", review site, reseller, etc. And another PITA is product review pages with SFA reviews.
netcan 17y ago
If having adsense ads reduced your search traffic, sites would just use some other ad service. Would you want to remove all sites using advertising of any kind? "Google bans advertising on all sites other than Google."
- jacquesm 17y ago
  I think that is mostly a matter of degree. You could conceivably make it switchable so that you only cut out certain advertising networks. MFA seems to be by far the larger part of the problem, possibly other ad networks would see an increase but nobody has the advertiser volume required for that other than adsense. Of course the final outcome of something like that remains to be seen but I'm fairly sure it would be a net positive. And they could always default it to 'off'.
  - stcredzero 17y ago
    Google could develop some kind of metric for "too many ads" on a page. Just getting the "O" in SEO to mean some sort of moderation would be a kind fo victory.
    - netcan 17y ago
      I wouldn't be surprised if they do, or at least have considered it. The problem is that would be undermining other businesses with advertising based business models, their direct & indirect competitors. Google don't want to send you to made-for-adsense sites. These account for a tiny portion of their income while degrading both their search results & their advertising offering. Google' two primary concerns.

m0th87 17y ago

Has the author forgotten how impossible it was to navigate the WWW prior to Google? They essentially fixed search relative to the competitive solutions at the time (Lycos, AltaVista...shudder).

dkasper 17y ago
Perhaps he's thinking things seem to be regressing back to those days.
dangrossman 17y ago
It's not Google search that's the problem, it's Google AdSense.
- jacquesm 17y ago
  Almost. It's the interaction between the two, google can't cut off the MFA sites from their serps without hurting their bottom line.
lmoorman 17y ago
That's like saying do you remember how hard it was to use a PC before Windows came around? The problems solved yesterday do not give you a pass today.
- Semiapies 17y ago
  Yes, but anyone saying, "Windows/OS X/Linux/your OS here destroyed the PC" is more blatantly trolling than the linked post.
- mattmcknight 17y ago
  It's actually not like saying that. He said Windows destroyed PCs.
GHFigs 17y ago
...and now it's broken again, for different reasons. These things happen in cycles.

Perceval 17y ago

It might interesting to try and design a reverse-search engine, a kind of internet spam filter. Instead of being designed to promote relevant content, it would be designed to explicitly not return crap content. Rather than trying to improve Google's resiliency against noise, it might be more fruitful to turn the problem around.

GHFigs 17y ago
The answer is people. People you trust to not provide you with crap. Sometimes that's friends, sometimes that's family, sometimes that's strangers. It's never social media webcocks who want to "friend you", never bloggers offering you "n Ways to X your Y" for their $5/mo. in AdSense earnings, and only rarely the nebulous "crowd".
- ramchip 17y ago
  Not so sure. When I'm looking for information on, say, WWII history, I can't ask someone or go on a trusted website like when I'm looking for information on a field I'm familiar with. I don't have any acquaintance or favourite websites already, which is why I'm using a search engine. The large amount of [quiz,farm,friend of the day] spam on my facebook, and the various hoax chain letters my not-too-web-savvy aunt sends us sometimes, also make me a little jaded on the value of 'social $whatever' for information gathering.
  - GHFigs 17y ago
    WWII history, I can't ask someone or go on a trusted website like when I'm looking for information on a field I'm familiar with. WWII history isn't such a hot topic for crapflooding, either. Search is still useful there. You're also still applying trust heuristics based on the results--"battle of the coral sea" returns results from Wikipedia, history.navy.mil, worldwar2history.info, history.sandiego.edu, etc. You probably have some idea about the content of each. Ultimately your decisions about what to click come down to your perception of trust in their authors' ability to provide you the information you desire. I'm just guessing, but I suspect most people here are familiar with the phenomenon of doing a Google search only to end up clicking on the Wikipedia result that invariably makes it into the top few results. This is a sign that Google's PageRank still aligns reasonably well with our own estimation of trust for those search terms. As for social whatevers: that's why I said "People you trust to not provide you with crap." You can't trust most people to do that. Least of all when given a megaphone and/or a financial incentive.

pmarsh 17y ago

If you want less page spam in your results write better queries into the search engine. There are still plenty of good results and it's fairly easy to ignore these SPAM pages just like you ignore banner ads. If the author said that Google's gotten worse then there might be reason to worry. But they're constantly improving I wouldn't worry about them getting beat at this game since they're the ones who control the rules.

stcredzero 17y ago

What he wants isn't a search engine that punishes ads. What he wants is really a search engine that punishes bad taste. (Specifically, too many ads greedily placed and greedily optimized text poorly written.) I think that such a search engine has tremendous profit potential. Would it be possible to detect redundancy? Is it possible to detect poor design?

10ren 17y ago
Wasn't he was focusing mainly on the result not answering his question, rather than taste (though both are factors). At first I was thinking that what you seek amounts to strong AI, but then I remembered that there are measures for objectively assessing how easy text is to understand (even Word has some built-in); though, ironically, text that is understandable by a lower age is usually easier to understand (i.e. better) than that which is so convoluted that it requires multiple PhDs to parse. Unfortunately, Google is not finding these metrics for me, because I don't know what they're called... ah, here we go: Passive Sentences Test, Flesch Reading Ease and Flesch-Kincaid Grade Level.
- stcredzero 17y ago
  I bet Google could do some simple metrics based on usefulness ratings by users (vote arrows) and number of ads. I bet they could also develop metrics to measure the same ratings against "degree of search engine optimization."

vaksel 17y ago

so how exactly does this guy propose content sites get paid if they can't use advertising to support themselves? Are they supposed to charge for access to read things that are readily available online?

dangrossman 17y ago
Real content sites aren't the problem, it's the sites that consist of nothing but thousands of pages of filler text and ads. A lot of them take articles off "free to reprint" article sites, and run them through programs that rearrange sentences and swap words with synonyms, to create "unique" content. Once Google finds the site, they start listing the pages, unable to differentiate between this garbage and real content.
- vaksel 17y ago
  except Google has a strict policy against sites like that, and smacks them down the second they find them.

ThinkWriteMute 17y ago

There’s an opportunity here to create a web search engine that punishes results littered with ads. Google can’t do it – they live off those ads. This is absolute bullshit and it looks like a majority of replies here are eating it up. Google actively de-lists pages that aren't content intensive, and I mean serious legible content. Research some SEO IM work before you try and suggest that Google does nothing to remove Ad-sense filled pages. It's this kind of blogging that really muddies the waters.

ThinkWriteMute 17y ago
Looks like he denied my post directly on the blog. Real big of him. False alarm, just slow to approve. In parting I'd like to say this: wake up, Ad-riddled sites get tossed up in miliseconds (I know, I've written an engine to do so). You can't possibly make a search engine that can de-list fast enough. SEO IMers will fucking break you and that's just the way it's going to be unless someone (God forbid) regulates who can make websites and how fast. post script, there are millions, if not hundreds of millions, of legit websites that use the easy and unobtrusive Ad-sense ads to pay for (sometimes) very expensive hosting. You want to talk about destroying the internet? You're describing the obliteration of (almost) any non-corporate website.

johnl 17y ago

I've noticed the same problem, on some searches the entire first page is pure junk. Bet 80% of internet users don't know the difference. Maybe a power users mode that drops those from the search results would work.

bobbyi 17y ago

Today I searched for the answer to a question. The top hit was a useful article written by a subject-matter expert. (Good job, Google.) Not really destroyed then.