8 ms·

Dear Google, I am writing an open letter from the search wilderness

41 points by sunnykgupta 12y ago

deleted 12y ago
[deleted]
natch 12y ago
TL;DR: Lamentably, Google has facilitated a transition to organic search results being strictly a money game for most categories, removing the joy of discovery and disproportionally impacting small businesses. What's missing is a suggestion of how to fix it. Not an easy problem.
- grey-area 12y ago
  I was wondering about this the other day, and considering another approach to search, which I'm sure others have thought of too as it's fairly obvious. To restate the problem: For any given search term there are millions of hits Of those hits, only the first 20 or so in a given ranking will ever be seen This renders a huge amount of the internet invisible and a huge amount of indexing/linking moot At present google uses what sounds like a very complex mix of signals about linking, reputation, novelty etc which it constantly tweaks, and delivers a relatively stable ranking which guarantees that those with deep pockets will win the game of ranking highest simply by buying top position or buying authority in various ways (sponsored blogs, buying authoritative domains, content farms etc). Another approach would be to constantly inject random search results for a given term into results, and use solely reader reactions to results to rank them (registered user voting, do they come back to search results or not, sites chosen as high quality by this user, that sort of signal). So always have some randomly chosen results to guarantee mixing, and always include some degree of user input to give better and more relevant results chosen by humans. I think you could build good results like this, while completely ignoring any sort of complex ranking algorithm which is both too stable, too prone to gaming, and in conflict with the revenue stream of ads. Obviously you'd have to be careful about which users you listened to, but that is a solvable problem if you can persuade people to register to get better results. It's definitely well past time for a few serious non-corporate competitors to google to emerge, if only to keep Google honest.
  - lingben 12y ago
    how would you track the user inputs in order to create then rank the results further? via browsers? or...? while I like your idea, I'm not sure if it is feasible and second, if there is an 'acceptable' method which most people will not object to or try to game. for example, imagine 1000's of mechanical turks being paid cents to click and browse to a specific link in order to fool the algorithm you've set up
    - grey-area 12y ago
      Tracking users and combatting fraud would definitely be an issue (just as it is for google), and you might have to set up trusted rings of users or some sort of web of trust rather than just trusting any user. It'd be interesting to try - tracking would have to be by browser cookie and actions like returning to results or upvoting/downvoting on the results page and you'd have to be very strict about fraud prevention from the start. Perhaps more interesting than a specific ranking algorithm though is the idea of injecting random results into the mix. Perhaps Google has tried all these things already, it wouldn't surprise me and I know they look at user reactions like returning to results/time on page etc, but it'd be nice to see some search engine which is not owned and controlled by a massive corporation like MS, Google or Apple though - is anyone doing something interesting with common crawl?
      - walterbell 12y ago
        There's a German metasearch engine funded by an NGO & University, which also funds the Yacy P2P engine: https://www.metager.de/en/ https://www.metager.de/en/ and http://yacy.net http://yacy.net Scholar Ninja is trying to use WebRTC for distributed indexing and searching from a Chrome extension, but there's a Chrome/WebRTC bug which prevents long-running tasks: http://juretriglav.si/an-open-distributed-search-engine-for-science/ http://juretriglav.si/an-open-distributed-search-engine-for-...
        grey-area 12y ago
        Thanks
- lazyjones 12y ago
  Not a solution, but a thought: it's easier to game a single search engine that owns the market than it would be to game 3 or 4 competing and different products with a significant market share each. Google's success has its downsides.
mgamache 12y ago
Okay. It's not Google's goal to provide the absolute best answer. They just want to avoid a wrong answer while providing an acceptable answer for a majority of searchers. Google has over-valued authority domains at the cost of smaller sites/blogs that have rich/deep/narrow content. Faking an authority domain is difficult using black-hat seo techniques so it's the sledge hammer that Google uses today. So yes, big sites are rewarded and the little guys are locked out if they try and compete.
evntik 12y ago
Gah! SEO posts are so tedious and annoying. Maybe if your business depends mostly on search traffic to survive it shouldn't. Further more publicly announcing that you must be entitled to it is embarrassing, specially when the plea is a thinly veiled "you used to be better" diatribe. If it was written as a bug report I would be more sympathetic, as it would identify specific problems, highlight desired behavior and suggest a solution, but more importantly it would be brief and to the point.
- anon1385 12y ago
  It's always fun to see Google employees admitting that yes Google only exists to extract as much wealth as possible from the rest of the world and doesn't give a single shit about anybody or anything else. If only you were able to bring the same enlightenment to some of your colleagues who post here.
- mgamache 12y ago
  The post is a bit winey, but it's a real problem. Google's search results are not much better then they were five years ago (worse in many cases). They are having a difficult time combating web-spam. Also, are you suggesting no business should try and create an ad supported website? To get the volume of viewers any public site needs organic search is a must.
lnanek2 12y ago
Don't really understand his point. Just because he is a little business and hates big brands doesn't mean users in general don't want big brands preferred in their search results. If someone is searching for a music video, yes, they most often mean the big brand currently at the top of the charts. If they are searching for electronics, yes BestBuy probably gives them better results than a tiny shop. Google does track clicks on results, so they do know which results the users actually prefer.
- Kalium 12y ago
  It's basically a long-form complaint that Google doesn't show users what he wants it to.
  - RichardDHH 12y ago
    Nope its not. It illustrates the divergence between Google mission statement based on technology and position and evoved reality.
    - Kalium 12y ago
      No, it really doesn't. It illustrates a changing business reality and your fond wish that things change in a way you happen to find convenient. You want google to provide not the thing that google and users think is most useful, but the things that YOU think is most useful. And incidentally (totally by accident, I'm sure) most profitable for you. Who do you think has more data and is in a better position to make that call - google or you?
- jccalhoun 12y ago
  his point is that his business isn't ranked as high on google as he thinks it should be.
  - bsenftner 12y ago
    His point is that none of the businesses in his industry can get a listing within their industry. His point is that intermediate "marketplaces" inject themselves into an industry and then control consumer access to that industry's products. These "marketplaces" are not actually in these industries - they are in the internet business aggregate industry, and serve the consumers very poorly, because after all the consumer is NOT their customer, their customer is their investors.
Kalium 12y ago
TL;DR: Google is giving users what they want instead of what I want, please change this. Seriously, what this guy wants is a search engine that promotes what he wants over what users want. He fails to understand that Google became popular specifically because it didn't do that.
- sunnykgupta 12y ago
  Sensible summarisation.
- asuffield 12y ago
  He goes a little further: he's objecting to the fact that Google is not acting as a "leveling" force to break up incumbent industries and change society. He appears to want them to act as a far-left government. I don't really feel a need to rebut this desire, beyond saying "no".
  - anon1385 12y ago
    It's kind of fascinating how the supposed economic and social benefits of the open web that people spent 20 years or more evangelising are dismissed as 'far-left' once powerful incumbents have cornered the market. Suddenly advocating for the destructive forces of open competition is the same as being a radical socialist.
    - Kalium 12y ago
      The problem here isn't that there isn't open competition. The problem is that he's lost at it.
      - pessimizer 12y ago
        The problem is that it isn't competition on services or products, it's competition in gaming and/or just paying massive incumbent middlemen.
        12y ago
jccalhoun 12y ago
"There is however one area of Google’s vast and growing empire that gives rise for serious concern and this is its foundation, “Google Search”, the most used and admired search engine on planet Earth and in fact, our solar system, unless Jupiter’s moons are hiding a quantum search tool." If this is how he always writes then no wonder his stuff isn't ranked higher. Eschew obfuscation!
sbierwagen 12y ago
Google doesn't owe you traffic.
higherpurpose 12y ago
I agree, Google has increasingly focused on big companies, ever since they announced the "focus on brands" 5 or so years ago.
us0r 12y ago
This guy is ranking pretty good for most of the circle jerk linking scheme on the bottom of his page (which violates Google's policy - LOL if they slap him).
aurumpotest 12y ago
And yet the images he used in his letter came from Shutterstock, rather than directly contacting a photographer and checking they're okay with their image being used to illustrate the point he's making. Not saying this isn't a problem, just that he doesn't like it when it brings him less money, yet he uses it in other industries because it's convenient. Having some way to filter out the big agglomerators would be nice - obviously you can use 'buy used car -autotrader', but the majority of users don't realise they can do that, even if they would like to.
- bhartzer 12y ago
  Images are good, but not needed to get his point across.
- RichardDHH 12y ago
  I am the author and there are no images, except a header at all in the original. They were added by TNOOZ. Original is here: http://open-letter-to-google.blogspot.co.uk/2014/08/open-letter-to-google-inc-cry-from.html http://open-letter-to-google.blogspot.co.uk/2014/08/open-let...
ohitsdom 12y ago
I don't think Google Search's goal was to ever help the user discover an unknown business. Its to give the user the best results for their search term (which, surprise, is the more popular businesses). Not strictly on topic, but a post like this is hard to read with so many exclamation points! Just make your dramatic point through logic, don't force it on the reader via punctuation!

41 points by sunnykgupta 12y ago

deleted 12y ago

[deleted]

natch 12y ago

TL;DR: Lamentably, Google has facilitated a transition to organic search results being strictly a money game for most categories, removing the joy of discovery and disproportionally impacting small businesses. What's missing is a suggestion of how to fix it. Not an easy problem.

grey-area 12y ago
I was wondering about this the other day, and considering another approach to search, which I'm sure others have thought of too as it's fairly obvious. To restate the problem: For any given search term there are millions of hits Of those hits, only the first 20 or so in a given ranking will ever be seen This renders a huge amount of the internet invisible and a huge amount of indexing/linking moot At present google uses what sounds like a very complex mix of signals about linking, reputation, novelty etc which it constantly tweaks, and delivers a relatively stable ranking which guarantees that those with deep pockets will win the game of ranking highest simply by buying top position or buying authority in various ways (sponsored blogs, buying authoritative domains, content farms etc). Another approach would be to constantly inject random search results for a given term into results, and use solely reader reactions to results to rank them (registered user voting, do they come back to search results or not, sites chosen as high quality by this user, that sort of signal). So always have some randomly chosen results to guarantee mixing, and always include some degree of user input to give better and more relevant results chosen by humans. I think you could build good results like this, while completely ignoring any sort of complex ranking algorithm which is both too stable, too prone to gaming, and in conflict with the revenue stream of ads. Obviously you'd have to be careful about which users you listened to, but that is a solvable problem if you can persuade people to register to get better results. It's definitely well past time for a few serious non-corporate competitors to google to emerge, if only to keep Google honest.
- lingben 12y ago
  how would you track the user inputs in order to create then rank the results further? via browsers? or...? while I like your idea, I'm not sure if it is feasible and second, if there is an 'acceptable' method which most people will not object to or try to game. for example, imagine 1000's of mechanical turks being paid cents to click and browse to a specific link in order to fool the algorithm you've set up
  - grey-area 12y ago
    Tracking users and combatting fraud would definitely be an issue (just as it is for google), and you might have to set up trusted rings of users or some sort of web of trust rather than just trusting any user. It'd be interesting to try - tracking would have to be by browser cookie and actions like returning to results or upvoting/downvoting on the results page and you'd have to be very strict about fraud prevention from the start. Perhaps more interesting than a specific ranking algorithm though is the idea of injecting random results into the mix. Perhaps Google has tried all these things already, it wouldn't surprise me and I know they look at user reactions like returning to results/time on page etc, but it'd be nice to see some search engine which is not owned and controlled by a massive corporation like MS, Google or Apple though - is anyone doing something interesting with common crawl?
    - walterbell 12y ago
      There's a German metasearch engine funded by an NGO & University, which also funds the Yacy P2P engine: https://www.metager.de/en/ https://www.metager.de/en/ and http://yacy.net http://yacy.net Scholar Ninja is trying to use WebRTC for distributed indexing and searching from a Chrome extension, but there's a Chrome/WebRTC bug which prevents long-running tasks: http://juretriglav.si/an-open-distributed-search-engine-for-science/ http://juretriglav.si/an-open-distributed-search-engine-for-...
      - grey-area 12y ago
        Thanks
lazyjones 12y ago
Not a solution, but a thought: it's easier to game a single search engine that owns the market than it would be to game 3 or 4 competing and different products with a significant market share each. Google's success has its downsides.

mgamache 12y ago

Okay. It's not Google's goal to provide the absolute best answer. They just want to avoid a wrong answer while providing an acceptable answer for a majority of searchers. Google has over-valued authority domains at the cost of smaller sites/blogs that have rich/deep/narrow content. Faking an authority domain is difficult using black-hat seo techniques so it's the sledge hammer that Google uses today. So yes, big sites are rewarded and the little guys are locked out if they try and compete.

evntik 12y ago

Gah! SEO posts are so tedious and annoying. Maybe if your business depends mostly on search traffic to survive it shouldn't. Further more publicly announcing that you must be entitled to it is embarrassing, specially when the plea is a thinly veiled "you used to be better" diatribe. If it was written as a bug report I would be more sympathetic, as it would identify specific problems, highlight desired behavior and suggest a solution, but more importantly it would be brief and to the point.

anon1385 12y ago
It's always fun to see Google employees admitting that yes Google only exists to extract as much wealth as possible from the rest of the world and doesn't give a single shit about anybody or anything else. If only you were able to bring the same enlightenment to some of your colleagues who post here.
mgamache 12y ago
The post is a bit winey, but it's a real problem. Google's search results are not much better then they were five years ago (worse in many cases). They are having a difficult time combating web-spam. Also, are you suggesting no business should try and create an ad supported website? To get the volume of viewers any public site needs organic search is a must.

lnanek2 12y ago

Don't really understand his point. Just because he is a little business and hates big brands doesn't mean users in general don't want big brands preferred in their search results. If someone is searching for a music video, yes, they most often mean the big brand currently at the top of the charts. If they are searching for electronics, yes BestBuy probably gives them better results than a tiny shop. Google does track clicks on results, so they do know which results the users actually prefer.

Kalium 12y ago
It's basically a long-form complaint that Google doesn't show users what he wants it to.
- RichardDHH 12y ago
  Nope its not. It illustrates the divergence between Google mission statement based on technology and position and evoved reality.
  - Kalium 12y ago
    No, it really doesn't. It illustrates a changing business reality and your fond wish that things change in a way you happen to find convenient. You want google to provide not the thing that google and users think is most useful, but the things that YOU think is most useful. And incidentally (totally by accident, I'm sure) most profitable for you. Who do you think has more data and is in a better position to make that call - google or you?
jccalhoun 12y ago
his point is that his business isn't ranked as high on google as he thinks it should be.
- bsenftner 12y ago
  His point is that none of the businesses in his industry can get a listing within their industry. His point is that intermediate "marketplaces" inject themselves into an industry and then control consumer access to that industry's products. These "marketplaces" are not actually in these industries - they are in the internet business aggregate industry, and serve the consumers very poorly, because after all the consumer is NOT their customer, their customer is their investors.

Kalium 12y ago

TL;DR: Google is giving users what they want instead of what I want, please change this. Seriously, what this guy wants is a search engine that promotes what he wants over what users want. He fails to understand that Google became popular specifically because it didn't do that.

sunnykgupta 12y ago
Sensible summarisation.
asuffield 12y ago
He goes a little further: he's objecting to the fact that Google is not acting as a "leveling" force to break up incumbent industries and change society. He appears to want them to act as a far-left government. I don't really feel a need to rebut this desire, beyond saying "no".
- anon1385 12y ago
  It's kind of fascinating how the supposed economic and social benefits of the open web that people spent 20 years or more evangelising are dismissed as 'far-left' once powerful incumbents have cornered the market. Suddenly advocating for the destructive forces of open competition is the same as being a radical socialist.
  - Kalium 12y ago
    The problem here isn't that there isn't open competition. The problem is that he's lost at it.
    - pessimizer 12y ago
      The problem is that it isn't competition on services or products, it's competition in gaming and/or just paying massive incumbent middlemen.
      - 12y ago

jccalhoun 12y ago

"There is however one area of Google’s vast and growing empire that gives rise for serious concern and this is its foundation, “Google Search”, the most used and admired search engine on planet Earth and in fact, our solar system, unless Jupiter’s moons are hiding a quantum search tool." If this is how he always writes then no wonder his stuff isn't ranked higher. Eschew obfuscation!

sbierwagen 12y ago

Google doesn't owe you traffic.

higherpurpose 12y ago

I agree, Google has increasingly focused on big companies, ever since they announced the "focus on brands" 5 or so years ago.

us0r 12y ago

This guy is ranking pretty good for most of the circle jerk linking scheme on the bottom of his page (which violates Google's policy - LOL if they slap him).

aurumpotest 12y ago

And yet the images he used in his letter came from Shutterstock, rather than directly contacting a photographer and checking they're okay with their image being used to illustrate the point he's making. Not saying this isn't a problem, just that he doesn't like it when it brings him less money, yet he uses it in other industries because it's convenient. Having some way to filter out the big agglomerators would be nice - obviously you can use 'buy used car -autotrader', but the majority of users don't realise they can do that, even if they would like to.

bhartzer 12y ago
Images are good, but not needed to get his point across.
RichardDHH 12y ago
I am the author and there are no images, except a header at all in the original. They were added by TNOOZ. Original is here: http://open-letter-to-google.blogspot.co.uk/2014/08/open-letter-to-google-inc-cry-from.html http://open-letter-to-google.blogspot.co.uk/2014/08/open-let...

ohitsdom 12y ago

I don't think Google Search's goal was to ever help the user discover an unknown business. Its to give the user the best results for their search term (which, surprise, is the more popular businesses). Not strictly on topic, but a post like this is hard to read with so many exclamation points! Just make your dramatic point through logic, don't force it on the reader via punctuation!