17 ms·

Google confirms the leaked Search documents are real

275 points by alanzhuly 2y ago

nioj 2y ago
Related discussion: https://news.ycombinator.com/item?id=40514491 https://news.ycombinator.com/item?id=40514491
- jader201 2y ago
  Which is a dupe of: https://news.ycombinator.com/item?id=40496967 https://news.ycombinator.com/item?id=40496967
jongjong 2y ago
Google should change their algorithm to rank websites randomly; they all show up in search results with equal probability, so long as they exceed a certain threshold of relevance for the user's keywords (the threshold could vary for different keywords but would be made public and there could be instructions on how to meet the threshold requirements so it doesn't have to be a secret and anyone should be able to get their sites showing for at least one set of specific keywords). That would make it impossible to game. Maybe they could have 5 slots in a side container for 'Top trending' for those keywords for the current day, week, month or year (the user can choose the granularity). Problem solved.
- groone 2y ago
  The golden times were 8-10 years ago, where you could change the order of keywords in google search and get more precise matches. Could find pretty much any obscure thing on the internet. Then could find that article that you remember read 6 months ago by adjusting the keywords until it is on the first page. Now it does not matter at all what you enter in the search box. No matter the inputs you get one set of results and will never find something specific.
- xyzzy123 2y ago
  You would game it by creating more websites.
  - jongjong 2y ago
    But would be difficult to build a lot of websites which all meet the threshold for specific keywords. The thresholds don't have to be particularly low. In fact it's better if they require a certain amount of work to meet. So maybe only a relatively small number of websites would qualify for a specific niche keyword but the idea is that, among those, they are ranked randomly. You'd probably have to use AI to figure out site quality in niche areas. Or Google could go with a lower risk approach of keeping their results as they are with their current algorithm, but only randomize 3 slots out of the top 10 based on this new threshold approach.
    - dmoy 2y ago
      Do you remember those autogenerated websites that were just giant lists of all words? Those disappeared many years ago, but if you made search ranking random, they'd come right back.
      - eastbound 2y ago
        In 2030: Do you remember those websites storytelling about their grand mother just to introduce a mathematical theorem? We’re so lucky they disappeared like the giant lists of all words, because they were 100% fabricated by Google’s unnatural incentives. Google has the ability to change the face of the internet in 2-3 years. They can detect the chaff and shut it down, and I wonder whether it’s an anti-competiton feature that they require that websites write a thousand words per page.
        Auracle 2y ago
        I asked ChatGPT to tell me how to get away with murder in the style of a recipe blog and it (surprisingly) did a bang-up job: https://chatgpt.com/share/b738b68d-8294-4a2c-87ff-f95a6e2d918f https://chatgpt.com/share/b738b68d-8294-4a2c-87ff-f95a6e2d91... I did this after simply wanting to know how much powdered sugar to put in whipped cream and getting frustrated at trying to scroll through 3 blogs just to find the ingredient list for something so simple. Eventually I just asked ChatGPT. I wonder if Google can start running an LLM on websites to judge them on things like that. Hell, looking for a photographer in your area? Have it judge how good the photography is on each website. The possibilities are there but I don’t know if they’ll bother.
- krisoft 2y ago
  > That would make it impossible to game. Have you considered that it would make it also pretty lame user experience?
  - amarcheschi 2y ago
    Hold up, I think he might be up to something for when he discovers to order an array in O(1)
- stephen_cagle 2y ago
  Yeah, and we should give it a cool name. Something that communicates that this is a new kind of search! I am thinking "NewHorizons" or how about "AltaVista"? What do you think?
jeroenhd 2y ago
I was afraid of this. Now, it's a matter of time before Google search will get even worse as SEO hustlers push more of their useless crap to the top now that internal algorithm data has been published. Guess I should look into that Kagi thing people keep mentioning.
- drivebycomment 2y ago
  It doesn't look like this leak will do that. This doesn't have "algorithms" in any real sense of algorithms.
  - Jensson 2y ago
    Before SEO communities argued what were part of the model, thinking some things like clicks didn't matter. Truth is Google did everything you can imagine, including clicks, now they all know it instead of having to guess. Still wont change much though, its very hard to game since Google has a lot of ways of mitigating click farms or it would have discovered a long time ago.
    - aaronwall 2y ago
      Back in the day a friend mentioned you could choose what version of a phrase you wanted to make the canonical for a search autocompletion by embedding a broken image call to a SERP page for the version of a keyword you wanted to be more popular. Google has tons of ways to identify real users versus fake users. And lots of the fake it until you make it efforts leave statistical outliers that can lead to ignoring or smoothing away much of the benefits, especially if there is no fire following the smoke trail.
- disagree123 2y ago
  [flagged]
  - regnerba 2y ago
    Care to expand upon that? I have been using Kagi for just under a year so far and really enjoying it.
blueflow 2y ago
10 years ago "chrome botnet" was a meme. And now we get the evidence for it. I don't know what to say.
beeboobaa3 2y ago
Any good summaries about what was revealed in the leak?
- janmo 2y ago
  The main takeaway for me is that Google is caught lying. Many things were already assumed but Google used to deny them. - They claimed that clicks were not a ranking factor, it turns out it is. - Also turns out that they are using Chrome data for ranking purposes (Not good for the ongoing lawsuit) - There is also a field called something like "is small personal site" and it presumed that those sites are penalized. You can find a summary here: https://www.reddit.com/r/SEO/comments/1d2gllz/google_caught_in_their_lies_with_leaked_api_docs/ https://www.reddit.com/r/SEO/comments/1d2gllz/google_caught_...
  - kobalsky 2y ago
    didn’t they just leak the schema? we know they may be tracking that information but we don’t know how it affects the model
    - _heimdall 2y ago
      Would they track the data if they haven't been using it, or didn't expect to use it in the future?
      - UncleMeat 2y ago
        It is hard to delete protobuf fields.
      - brokenmachine 2y ago
        Didn't they just say they're not using it currently?
      - deleted 2y ago
        [deleted]
janmo 2y ago
The main takeaway is that Google has been lying and gaslighting about their ranking factors. The main lies that were uncovered is that they are indeed using clicks, and chrome browser data for ranking purposes. Summary of their lies here: https://www.reddit.com/r/SEO/comments/1d2gllz/google_caught_in_their_lies_with_leaked_api_docs/ https://www.reddit.com/r/SEO/comments/1d2gllz/google_caught_...
- pixxel 2y ago
  https://old.reddit.com/r/SEO/comments/1d2gllz/google_caught_in_their_lies_with_leaked_api_docs/ https://old.reddit.com/r/SEO/comments/1d2gllz/google_caught_...
- _heimdall 2y ago
  I don't understand why anyone would trust those lies though. For a very long time Google has been a data and advertising business with a monopolistic hold on browsers, search, analytics, and advertising. Of course they use those together to make more money.
rexreed 2y ago
In not too many years, the average user will be prompting to get their needs met, rather than searching a flawed search system, wading through pages of sponsored and SEO-gamified links, opening up multiple tabs to try to dig out the details from sites hustling whatever they hustle, and then trying to read to get their needs met. Google sees the writing on the wall, which is going towards a prompt-based direct ask system mediated by an LLM. It definitely is far from perfect now, but the writing is on the wall. Search and SEO are both going to be relics of a bygone past in not too many years.
- mu53 2y ago
  I don't know. There were recently some documents released on how OpenAI was soliciting companies to integrate their product recommendations more deeply into the training data. This is obvious a huge way to monetize chatgpt-like products. Rather than SEO optimized sites gathering the ad revenue with click bait and gamification, OpenAI will collect the revenue themselves. At the end of this enshittification, users will be looking for other options. Imagine a salesman that is ignoring the elephant in the room to tell you that if you bought this brand of shoes, you would run faster instead of giving tips on the skills to learn to be a better runner. Search is a great way to find high-quality references and non-hallucinated answers by tweaking the keywords slightly. A salesman-like LLM might be pushing products when you just need information. ChatGPT's authority is going to dwindle, and search is a good tool to find authoritative sources
  - _heimdall 2y ago
    I'd argue that sales is entirely a game of ignoring the elephant in the room and selling someone on something they don't already think they need. Its not really sales if I go into a shoe store and say I want a pair of Air Jordan 4s in size 11, that's just customer service.
  - yoyogogo69 2y ago
    people already know how to be a better runner. they just dont want to do it. they rather buy shoes that make them run faster instead. people also dont like choice. they might think they do but really i dont believe it. getting a single answer with a simple question is more appealing than having to come up with a detailed question with many answers to choose from. even searches have been doing this with the single boxed result at the top
Awtem 2y ago
Good encryption should still hold, even if you know the algorithm. The same reasoning should be applied to search engines.
- Spivak 2y ago
  Oh there's no way such a thing is possible. Unless you have an omniscient oracle your search engine will be based on some metric correlated with quality and relevancy. The moment those are known people will produce low quality content with high scores on those metrics. Can you think of even a single metric that can't be gamed?
Dxtros 2y ago
to get real answers from humans and not marketing campaigns I constantly put reddit in my search. I feel like 5 years ago that wasn’t necessary.
- whatamidoingyo 2y ago
  > not marketing campaigns > reddit A great marketing tactic is to pose as reddit users. If you have just 2-3 realistic accounts, you can ask a question as account 1, and write your answer with account 2. Now imagine a company with $$$. They can guide an entire thread.
  - yoyogogo69 2y ago
    now imagine entire countries spreading influence and manipulation through reddit. add in extreme bias and hivemind. only a select few topics where reddit would be a good place to learn from
  - sdwr 2y ago
    I'm sure there are marketers on Reddit, but the nature of virality itself makes it pretty robust against attempts at manipulation. (You have to astroturf really hard or be a part of an existing wave, astroturfing reads different, so the best you can hope for is bending the narrative a step or two)
- aaronwall 2y ago
  Very easy to post things on Reddit as a marketer, particularly when working with a small group who can respond to each other to season threads. Plus you can pay trusted Reddit account holders to post items for you.
nojvek 2y ago
KEY TAKEAWAYS: • Google claimed they don't use a "domain authority" metric, but the docs show they totally do - it's called "siteAuthority." • G said clicks don't affect rankings, but there's a whole system called "NavBoost" that uses click data to change search results. • Google denied having a "sandbox" that holds back new sites, but yep, the docs confirm it exists. • G assured us Chrome data isn't used for ranking, but surprise! It is. • The number and diversity of your backlinks still matter a lot. • Having authors with expertise and authority helps. • Putting keywords in your title tag and matching search queries is important. • Google tracks the dates on your pages to determine freshness. • A lot of long-held SEO theories have been validated, so trust your instincts. • Creating great content and promoting it well is still the best approach. • We should experiment more to see what works, rather than just listening to what Google says. From: https://www.reddit.com/r/SEO/s/ChlTrhjPnG https://www.reddit.com/r/SEO/s/ChlTrhjPnG I wonder how chrome data works. Are they using every chrome browser to sniff what users are clicking on?
flembat 2y ago
Its getting harder and harder to like Google, not only are they NOT not evil, they are also not even competent.
Fer24 2y ago
Well, another reason to take whit caution what Google says

275 points by alanzhuly 2y ago

nioj 2y ago

Related discussion: https://news.ycombinator.com/item?id=40514491 https://news.ycombinator.com/item?id=40514491

jader201 2y ago
Which is a dupe of: https://news.ycombinator.com/item?id=40496967 https://news.ycombinator.com/item?id=40496967

jongjong 2y ago

Google should change their algorithm to rank websites randomly; they all show up in search results with equal probability, so long as they exceed a certain threshold of relevance for the user's keywords (the threshold could vary for different keywords but would be made public and there could be instructions on how to meet the threshold requirements so it doesn't have to be a secret and anyone should be able to get their sites showing for at least one set of specific keywords). That would make it impossible to game. Maybe they could have 5 slots in a side container for 'Top trending' for those keywords for the current day, week, month or year (the user can choose the granularity). Problem solved.

groone 2y ago
The golden times were 8-10 years ago, where you could change the order of keywords in google search and get more precise matches. Could find pretty much any obscure thing on the internet. Then could find that article that you remember read 6 months ago by adjusting the keywords until it is on the first page. Now it does not matter at all what you enter in the search box. No matter the inputs you get one set of results and will never find something specific.
xyzzy123 2y ago
You would game it by creating more websites.
- jongjong 2y ago
  But would be difficult to build a lot of websites which all meet the threshold for specific keywords. The thresholds don't have to be particularly low. In fact it's better if they require a certain amount of work to meet. So maybe only a relatively small number of websites would qualify for a specific niche keyword but the idea is that, among those, they are ranked randomly. You'd probably have to use AI to figure out site quality in niche areas. Or Google could go with a lower risk approach of keeping their results as they are with their current algorithm, but only randomize 3 slots out of the top 10 based on this new threshold approach.
  - dmoy 2y ago
    Do you remember those autogenerated websites that were just giant lists of all words? Those disappeared many years ago, but if you made search ranking random, they'd come right back.
    - eastbound 2y ago
      In 2030: Do you remember those websites storytelling about their grand mother just to introduce a mathematical theorem? We’re so lucky they disappeared like the giant lists of all words, because they were 100% fabricated by Google’s unnatural incentives. Google has the ability to change the face of the internet in 2-3 years. They can detect the chaff and shut it down, and I wonder whether it’s an anti-competiton feature that they require that websites write a thousand words per page.
      - Auracle 2y ago
        I asked ChatGPT to tell me how to get away with murder in the style of a recipe blog and it (surprisingly) did a bang-up job: https://chatgpt.com/share/b738b68d-8294-4a2c-87ff-f95a6e2d918f https://chatgpt.com/share/b738b68d-8294-4a2c-87ff-f95a6e2d91... I did this after simply wanting to know how much powdered sugar to put in whipped cream and getting frustrated at trying to scroll through 3 blogs just to find the ingredient list for something so simple. Eventually I just asked ChatGPT. I wonder if Google can start running an LLM on websites to judge them on things like that. Hell, looking for a photographer in your area? Have it judge how good the photography is on each website. The possibilities are there but I don’t know if they’ll bother.
krisoft 2y ago
> That would make it impossible to game. Have you considered that it would make it also pretty lame user experience?
- amarcheschi 2y ago
  Hold up, I think he might be up to something for when he discovers to order an array in O(1)
stephen_cagle 2y ago
Yeah, and we should give it a cool name. Something that communicates that this is a new kind of search! I am thinking "NewHorizons" or how about "AltaVista"? What do you think?

jeroenhd 2y ago

I was afraid of this. Now, it's a matter of time before Google search will get even worse as SEO hustlers push more of their useless crap to the top now that internal algorithm data has been published. Guess I should look into that Kagi thing people keep mentioning.

drivebycomment 2y ago
It doesn't look like this leak will do that. This doesn't have "algorithms" in any real sense of algorithms.
- Jensson 2y ago
  Before SEO communities argued what were part of the model, thinking some things like clicks didn't matter. Truth is Google did everything you can imagine, including clicks, now they all know it instead of having to guess. Still wont change much though, its very hard to game since Google has a lot of ways of mitigating click farms or it would have discovered a long time ago.
  - aaronwall 2y ago
    Back in the day a friend mentioned you could choose what version of a phrase you wanted to make the canonical for a search autocompletion by embedding a broken image call to a SERP page for the version of a keyword you wanted to be more popular. Google has tons of ways to identify real users versus fake users. And lots of the fake it until you make it efforts leave statistical outliers that can lead to ignoring or smoothing away much of the benefits, especially if there is no fire following the smoke trail.
disagree123 2y ago
[flagged]
- regnerba 2y ago
  Care to expand upon that? I have been using Kagi for just under a year so far and really enjoying it.

blueflow 2y ago

10 years ago "chrome botnet" was a meme. And now we get the evidence for it. I don't know what to say.

beeboobaa3 2y ago

Any good summaries about what was revealed in the leak?

janmo 2y ago
The main takeaway for me is that Google is caught lying. Many things were already assumed but Google used to deny them. - They claimed that clicks were not a ranking factor, it turns out it is. - Also turns out that they are using Chrome data for ranking purposes (Not good for the ongoing lawsuit) - There is also a field called something like "is small personal site" and it presumed that those sites are penalized. You can find a summary here: https://www.reddit.com/r/SEO/comments/1d2gllz/google_caught_in_their_lies_with_leaked_api_docs/ https://www.reddit.com/r/SEO/comments/1d2gllz/google_caught_...
- kobalsky 2y ago
  didn’t they just leak the schema? we know they may be tracking that information but we don’t know how it affects the model
  - _heimdall 2y ago
    Would they track the data if they haven't been using it, or didn't expect to use it in the future?
    - UncleMeat 2y ago
      It is hard to delete protobuf fields.
    - brokenmachine 2y ago
      Didn't they just say they're not using it currently?
    - deleted 2y ago
      [deleted]

janmo 2y ago

The main takeaway is that Google has been lying and gaslighting about their ranking factors. The main lies that were uncovered is that they are indeed using clicks, and chrome browser data for ranking purposes. Summary of their lies here: https://www.reddit.com/r/SEO/comments/1d2gllz/google_caught_in_their_lies_with_leaked_api_docs/ https://www.reddit.com/r/SEO/comments/1d2gllz/google_caught_...

pixxel 2y ago
https://old.reddit.com/r/SEO/comments/1d2gllz/google_caught_in_their_lies_with_leaked_api_docs/ https://old.reddit.com/r/SEO/comments/1d2gllz/google_caught_...
_heimdall 2y ago
I don't understand why anyone would trust those lies though. For a very long time Google has been a data and advertising business with a monopolistic hold on browsers, search, analytics, and advertising. Of course they use those together to make more money.

rexreed 2y ago

In not too many years, the average user will be prompting to get their needs met, rather than searching a flawed search system, wading through pages of sponsored and SEO-gamified links, opening up multiple tabs to try to dig out the details from sites hustling whatever they hustle, and then trying to read to get their needs met. Google sees the writing on the wall, which is going towards a prompt-based direct ask system mediated by an LLM. It definitely is far from perfect now, but the writing is on the wall. Search and SEO are both going to be relics of a bygone past in not too many years.

mu53 2y ago
I don't know. There were recently some documents released on how OpenAI was soliciting companies to integrate their product recommendations more deeply into the training data. This is obvious a huge way to monetize chatgpt-like products. Rather than SEO optimized sites gathering the ad revenue with click bait and gamification, OpenAI will collect the revenue themselves. At the end of this enshittification, users will be looking for other options. Imagine a salesman that is ignoring the elephant in the room to tell you that if you bought this brand of shoes, you would run faster instead of giving tips on the skills to learn to be a better runner. Search is a great way to find high-quality references and non-hallucinated answers by tweaking the keywords slightly. A salesman-like LLM might be pushing products when you just need information. ChatGPT's authority is going to dwindle, and search is a good tool to find authoritative sources
- _heimdall 2y ago
  I'd argue that sales is entirely a game of ignoring the elephant in the room and selling someone on something they don't already think they need. Its not really sales if I go into a shoe store and say I want a pair of Air Jordan 4s in size 11, that's just customer service.
- yoyogogo69 2y ago
  people already know how to be a better runner. they just dont want to do it. they rather buy shoes that make them run faster instead. people also dont like choice. they might think they do but really i dont believe it. getting a single answer with a simple question is more appealing than having to come up with a detailed question with many answers to choose from. even searches have been doing this with the single boxed result at the top

Awtem 2y ago

Good encryption should still hold, even if you know the algorithm. The same reasoning should be applied to search engines.

Spivak 2y ago
Oh there's no way such a thing is possible. Unless you have an omniscient oracle your search engine will be based on some metric correlated with quality and relevancy. The moment those are known people will produce low quality content with high scores on those metrics. Can you think of even a single metric that can't be gamed?

Dxtros 2y ago

to get real answers from humans and not marketing campaigns I constantly put reddit in my search. I feel like 5 years ago that wasn’t necessary.

whatamidoingyo 2y ago
> not marketing campaigns > reddit A great marketing tactic is to pose as reddit users. If you have just 2-3 realistic accounts, you can ask a question as account 1, and write your answer with account 2. Now imagine a company with $$$. They can guide an entire thread.
- yoyogogo69 2y ago
  now imagine entire countries spreading influence and manipulation through reddit. add in extreme bias and hivemind. only a select few topics where reddit would be a good place to learn from
- sdwr 2y ago
  I'm sure there are marketers on Reddit, but the nature of virality itself makes it pretty robust against attempts at manipulation. (You have to astroturf really hard or be a part of an existing wave, astroturfing reads different, so the best you can hope for is bending the narrative a step or two)
aaronwall 2y ago
Very easy to post things on Reddit as a marketer, particularly when working with a small group who can respond to each other to season threads. Plus you can pay trusted Reddit account holders to post items for you.

nojvek 2y ago

KEY TAKEAWAYS: • Google claimed they don't use a "domain authority" metric, but the docs show they totally do - it's called "siteAuthority." • G said clicks don't affect rankings, but there's a whole system called "NavBoost" that uses click data to change search results. • Google denied having a "sandbox" that holds back new sites, but yep, the docs confirm it exists. • G assured us Chrome data isn't used for ranking, but surprise! It is. • The number and diversity of your backlinks still matter a lot. • Having authors with expertise and authority helps. • Putting keywords in your title tag and matching search queries is important. • Google tracks the dates on your pages to determine freshness. • A lot of long-held SEO theories have been validated, so trust your instincts. • Creating great content and promoting it well is still the best approach. • We should experiment more to see what works, rather than just listening to what Google says. From: https://www.reddit.com/r/SEO/s/ChlTrhjPnG https://www.reddit.com/r/SEO/s/ChlTrhjPnG I wonder how chrome data works. Are they using every chrome browser to sniff what users are clicking on?

flembat 2y ago

Its getting harder and harder to like Google, not only are they NOT not evil, they are also not even competent.

Fer24 2y ago

Well, another reason to take whit caution what Google says