9 ms·

Yandex ‘leak’ reveals search ranking factors

278 points by Password2 4y ago

danpalmer 4y ago
From the article, the interesting factors seem to be things like number of unique visitors and percentage of organic traffic. The latter is interesting as a factor because it suggests that visitors are visiting for the value rather than because they've been targeted by ads, implying higher value. Not surprised that some websites have an artificial preference though. It feels like this is basically the selling point (other than privacy) of Kagi, DDG with its bang commands, and other alternative search engines – artificial inflation of rank.
- Password2 4y ago
  Yep, would be interesting to test if the "percentage of organic"-factor also is relevant in Google. This would mean that they penalize pages for buying their own product.
- ajsnigrutin 4y ago
  I find it interesting, that even though many people click on a google link to some page (without checking the url), see the page, in two seconds click back, and choose a different result, that all that doesn't downgrade that pages page rank (if a user (many of them) comes back to search results in two seconds, the content was obviously garbage). Some pages, that I won't name, but one of them starts with a pin- and ends with -terest consistently get high google results while offering garbage content.
  - danpalmer 4y ago
    It's possible that is a factor, but that some sites are just very good at boosting the other metrics, or that they provide some other kind of perceived value and are therefore up-ranked.
  - edgineer 4y ago
    Do Google et al really not check for that?
    - londons_explore 4y ago
      Of course they do. In fact, a good way to downrank a competitors pages is to do a search for them, click the result, then click back 5s later. Repeat 1000x from different google accounts. A week later, the competitor will go off the first page.
      - renonce 4y ago
        Did you try that in practice?
    - thrwyndx 4y ago
      They absolutely do
  - luckylion 4y ago
    Some people believe that it does, that's why they pay users to search for something and click through the serps & go back until they arrive at the target domain where they don't click back but stop the experiment, signalling that this page answered their question / solved their problem. I've seen some data on this, and that data was inconclusive. Sometimes it seems to work, sometimes it seems to be counter-productive, sometimes nothing happens. It's hard to truly test things in isolation, and maybe their methods were shit, but I don't think it really does anything, or maybe Google has solid ways to detect anomalies and ignore them.
  - kryptiskt 4y ago
    I often open a bunch of the results in new tabs. Which could be interpreted as I'm unhappy with the first ones I open, but I haven't even looked at them at that point.
    - ajsnigrutin 4y ago
      Well sure, but i'm sure google can detect if you clicked 5 results in five tabs, or if you've clicked on one, and clicked "back" immeditaly... if a huge amount of users stay only 2-3 seconds on a specific page and then come back, that would be a red flag for me. On the other hand, maybe some people actually like the pinterest pages in results, and i'm the weird one... who knows.
      - autoexec 4y ago
        > On the other hand, maybe some people actually like the pinterest pages in results, and i'm the weird one... who knows. It isn't just you! I suspect that for certain people it's easy for them to get distracted there though.
  - flakeoil 4y ago
    > ...comes back to search results in two seconds, the content was obviously garbage I don't think that if a user stays a short time necessarily means the content is garbage. It might also mean that the user found directly what he was looking for, such as the answer to the question. It might even mean the page is very good as it is clear and provides the answer and does not have other click-bait type content. Landing on a bad page, might require you to read and browse around and maybe after a while figure out it's garbage or you get distracted by some "interesting", but unrelated articles. But if the search engine would rank it high because of this then that would be wrong.
    - ajsnigrutin 4y ago
      This would be true, if a user then did another search for something else, or close the tab. If you got an answer, you usually don't come back and click the next link in the results.
      - Loic 4y ago
        For me the pattern changed in the past few years. Before, I would take the first good answer as the answer, they used to be of good quality. Now, I tend to look at a couple (up to 4) of results to kind of cross-validate the answer before using it. Sometimes it takes me quite some "click through the results pages" to find 4 diverse enough results to validate the answer. But looking at my kids, looking directly at the info box and considering the answer as given by God, I suppose I am an exception.
    - vidarh 4y ago
      If a user clicks through and don't end up back at the search engine, there is probably a high probability they found the answer whether or not they spent a short or long time at the page. But if they click through and then two seconds later comes back to the search engine results, that seems unlikely.
      - autoexec 4y ago
        I tend to scroll down the first few pages of results and open anything that appears interesting even if it's not really relevant to what I wanted. Hopefully that doesn't look similar enough that it might impact the ratings of sites on other search engines. It's possible that the first few results got me to what I was looking for (that's far from a given these days though) but I'm clicking on a lot of other results in a short amount of time regardless.
  - thrwyndx 4y ago
    Well, why would it downgrade pagerank which is based on web graph connectivity? You're not changing the graph in any way with your click, no? But Google definitely tracks such short clicks and considers them a bad signal, very important in ranking. Same for Yandex, search for "dwell time" (visit duration) in its code/factors.
- deleted 4y ago
  [deleted]
oefrha 4y ago
Apparently this leak is a huge boon to SEO. Expect more SERPs to become SEO-overtaken garbage dumps in the near future. Truly depressing.
- ad404b8a372f2b9 4y ago
  I'm doubtful this will have much of an effect for three reasons: - These are factors, not the underlying algorithms which use them, which means it's not possible to deduce your own ranking from that data. - Many of those are so vague as to be useless. - The most important factors for SEO are those we already know about.
  - zexodus 4y ago
    It's possible to use an algorithmic approach (i.e. linear regression) to derive the weights for each of these factors.
    - toto444 4y ago
      All you need to do is scrape the whole internet beforehand.
    - thrwyndx 4y ago
      A lot of these factors depend on internal yandex data, serp clicks especially - good luck getting those outside of yandex prod, and those are the strongest signals.
    - abm53 4y ago
      Given some samples of search results we still don't know: - The X matrix (e.g. the page rank score) for each result. - The y vector, i.e the score for each result. Although we can observe the relative ranking in each sample (would be interested to hear about techniques to cope with this).
    - michaelmior 4y ago
      This is assuming the weights are combined via some linear function. That's not necessarily the case. It's likely that the ranking algorithm is more complex than: calculate features, multiply by weights, add to get ranking. Sure you'll probably still be able to learn something more by playing with the different features, but I doubt you'll get any real meaningful "weights." Especially when how many features are calculated is a black box.
yodsanklai 4y ago
Could someone explain to me how you go from these ranking facts to an actual sorted list of results? and doing it with an acceptable latency. Also, what happened to Google page rank, is is still relevant today?
- fsflover 4y ago
  I can't answer, but you should be able to find the answer on https://yacy.net https://yacy.net -- free web search. By the way, it can probably tremendously benefit from the leak.
- miohtama 4y ago
  For an exhaustive answer, you can check from Yandex source code that was posted on another thread on HN https://news.ycombinator.com/item?id=34525936 https://news.ycombinator.com/item?id=34525936
- ad404b8a372f2b9 4y ago
  "SEOs have already started analyzing Yandex's search ranking factors, which include PageRank and several other link-related factors" PageRank is factor 0.
- WinLychee 4y ago
  All of the listed "facts" or "features" can be incorporated into an index (e.g. with Lucene) or fed into ML-models (logistic regression, trees, neural nets, etc). Without going into specifics, I've seen ranking treated as a multi-stage algorithmic problem. Initially, you rank results with a lower-quality but low-latency ranker at the first stage (inverted index, tf-idf, knn, etc), and subsequent stages rerank the top-K results with higher-latency ML models outputting a relevance score. I believe progress is currently being made to combine everything into one giant neural model that just ranks everything from the get-go rather than pass in multiple stages.
- ValleZ 4y ago
  Pages are sorted by page rank, you pick first 10-100k pages and sort using other factors, then take first 10-100 results and sort using most costly factors.
yamrzou 4y ago
The list of factors: https://yandex-explorer.herokuapp.com/search?q=&o=all https://yandex-explorer.herokuapp.com/search?q=&o=all
nicbvs 4y ago
95 - Ukrainian > It is equal to one if the site has a Ukrainian geoist (i.e. 1 - Ukrainian site)
- SXX 4y ago
  This is completely unrelated to recent events. Before initial invasion in 2014 many Russian IT companies had huge presence in Ukraine and many Yandex services including search had localized versions and own search ranking, etc. Yandex still have versions for Belarus, Kazahstan, other ex-USSR countries and even Turkey. But chances for international expansion was pretty much destroyd by political situation. Also keep in mind Yandex was different company a decade ago. Back then Ilya Segalovich been CTO there (he died of cancer in 2013) and he supported opposition and even participated in street protests in Moscow.
evolextra 4y ago
[flagged]
thekyle 4y ago
It's interesting that they consider the presence of ads and the presence of Yandex ads as separate factors. I wonder if this implies a priority for pages with Yandex ads.
- liveoneggs 4y ago
  at least we can be sure google would never consider such a ranking
  - bitxbitxbitcoin 4y ago
    Really? /S it’s the only reason why I have Adsense.
  - dotancohen 4y ago
    That would explain why some content farms rank about e.g. Stack Overflow and Wikipedia with content scraped from those sites.
    - deleted 4y ago
      [deleted]
RugnirViking 4y ago
Interesting ones I have found: 108 MusicQ: The musicality of the request. The results of the sorcerer Anton Konygin. 276 NightQuery: The request is set mainly at night (and accompanying ones for morning, afternoon etc) 316 IsUa: Domain in the .ua zone (ukraine) 747 WordHostWikiSum: The relative popularity of the Word -Host pair, where Word is the word from the Title article on Wikipedia, and the Host is the host that is referred to in this article. 755 NastyContent: Content ugliness factor. (several of the other factors related to port and "adult" content are defined as multiples of this) 892 ShortVideo: A document is a short video (Tiktok, Reels, Shorts) 894 TelegramPost: Document - post in telegram (note I could not find similar for twitter etc, but they do have for tiktok) 1076 More90SecVisitsShare: The share of visits for which the time spent during the day on the host is more than 90 seconds (they track how long users are spending on pages) 1078 RankHackedNovaPhp: Rank of hacked sites 1309 DistanceToAnkara: The distance from the city from where the request was set to Ankara 1603 YellowImgCount: Average yellow images count on host 1614 NewsAgencyRating: Rating of news agency from agencies.json (Yandex.News resource)
- inerte 4y ago
  Everything in there makes sense imho except the Yellow Image Count. Anyone knows why that would be a factor?
  - encoderer 4y ago
    No, but can you explain the distance to Ankara one?
    - inerte 4y ago
      well now you're making me doubt myself, that's a weird one too. My first guess was "they need to know how far one request is from another, and Ankara is a good as any city" So say suppose you're in New Orleans and I am in San Jose, now we can compare our distance, because geographical distance usually equates to cultural differences and perceptions of quality.
g_sch 4y ago
Maybe I'm just a newbie when it comes to building search algorithms, but how on earth would you maintain or test something like this? From experience, I know that search algorithms using far fewer ranking factors than this are deployed in production and treated as terrifying black boxes that no one wants to touch lest they break something in an unexpected way.
- thrwyndx 4y ago
  Testing is easy with any half decent development setup. You should have some train/eval datasets and monitor metrics on them during training, this is ML 101. And do live A/B experiments for launch candidates. Maintenance sure is hell, lots of sweat and tears. Just glancing over search/formula/webcommon/select_ranking_models.cpp makes me cringe, they must have many dozens if not hundreds of different models in prod by now. Each of them needing maintenance and lots of training data. Work on new ranking factors I suspect must be also highly frustrating: throwing stuff at wall^W catboost black box and seeing if it sticks, and if it doesn't you'd have little idea why and control over it. Imho google's approach (white-boxish interpretable top level ranking formulas) is far superior and maintainable at scale.

278 points by Password2 4y ago

From the article, the interesting factors seem to be things like number of unique visitors and percentage of organic traffic. The latter is interesting as a factor because it suggests that visitors are visiting for the value rather than because they've been targeted by ads, implying higher value. Not surprised that some websites have an artificial preference though. It feels like this is basically the selling point (other than privacy) of Kagi, DDG with its bang commands, and other alternative search engines – artificial inflation of rank.

Password2 4y ago
Yep, would be interesting to test if the "percentage of organic"-factor also is relevant in Google. This would mean that they penalize pages for buying their own product.
ajsnigrutin 4y ago
I find it interesting, that even though many people click on a google link to some page (without checking the url), see the page, in two seconds click back, and choose a different result, that all that doesn't downgrade that pages page rank (if a user (many of them) comes back to search results in two seconds, the content was obviously garbage). Some pages, that I won't name, but one of them starts with a pin- and ends with -terest consistently get high google results while offering garbage content.
- danpalmer 4y ago
  It's possible that is a factor, but that some sites are just very good at boosting the other metrics, or that they provide some other kind of perceived value and are therefore up-ranked.
- edgineer 4y ago
  Do Google et al really not check for that?
  - londons_explore 4y ago
    Of course they do. In fact, a good way to downrank a competitors pages is to do a search for them, click the result, then click back 5s later. Repeat 1000x from different google accounts. A week later, the competitor will go off the first page.
    - renonce 4y ago
      Did you try that in practice?
  - thrwyndx 4y ago
    They absolutely do
- luckylion 4y ago
  Some people believe that it does, that's why they pay users to search for something and click through the serps & go back until they arrive at the target domain where they don't click back but stop the experiment, signalling that this page answered their question / solved their problem. I've seen some data on this, and that data was inconclusive. Sometimes it seems to work, sometimes it seems to be counter-productive, sometimes nothing happens. It's hard to truly test things in isolation, and maybe their methods were shit, but I don't think it really does anything, or maybe Google has solid ways to detect anomalies and ignore them.
- kryptiskt 4y ago
  I often open a bunch of the results in new tabs. Which could be interpreted as I'm unhappy with the first ones I open, but I haven't even looked at them at that point.
  - ajsnigrutin 4y ago
    Well sure, but i'm sure google can detect if you clicked 5 results in five tabs, or if you've clicked on one, and clicked "back" immeditaly... if a huge amount of users stay only 2-3 seconds on a specific page and then come back, that would be a red flag for me. On the other hand, maybe some people actually like the pinterest pages in results, and i'm the weird one... who knows.
    - autoexec 4y ago
      > On the other hand, maybe some people actually like the pinterest pages in results, and i'm the weird one... who knows. It isn't just you! I suspect that for certain people it's easy for them to get distracted there though.
- flakeoil 4y ago
  > ...comes back to search results in two seconds, the content was obviously garbage I don't think that if a user stays a short time necessarily means the content is garbage. It might also mean that the user found directly what he was looking for, such as the answer to the question. It might even mean the page is very good as it is clear and provides the answer and does not have other click-bait type content. Landing on a bad page, might require you to read and browse around and maybe after a while figure out it's garbage or you get distracted by some "interesting", but unrelated articles. But if the search engine would rank it high because of this then that would be wrong.
  - ajsnigrutin 4y ago
    This would be true, if a user then did another search for something else, or close the tab. If you got an answer, you usually don't come back and click the next link in the results.
    - Loic 4y ago
      For me the pattern changed in the past few years. Before, I would take the first good answer as the answer, they used to be of good quality. Now, I tend to look at a couple (up to 4) of results to kind of cross-validate the answer before using it. Sometimes it takes me quite some "click through the results pages" to find 4 diverse enough results to validate the answer. But looking at my kids, looking directly at the info box and considering the answer as given by God, I suppose I am an exception.
  - vidarh 4y ago
    If a user clicks through and don't end up back at the search engine, there is probably a high probability they found the answer whether or not they spent a short or long time at the page. But if they click through and then two seconds later comes back to the search engine results, that seems unlikely.
    - autoexec 4y ago
      I tend to scroll down the first few pages of results and open anything that appears interesting even if it's not really relevant to what I wanted. Hopefully that doesn't look similar enough that it might impact the ratings of sites on other search engines. It's possible that the first few results got me to what I was looking for (that's far from a given these days though) but I'm clicking on a lot of other results in a short amount of time regardless.
- thrwyndx 4y ago
  Well, why would it downgrade pagerank which is based on web graph connectivity? You're not changing the graph in any way with your click, no? But Google definitely tracks such short clicks and considers them a bad signal, very important in ranking. Same for Yandex, search for "dwell time" (visit duration) in its code/factors.
deleted 4y ago
[deleted]

oefrha 4y ago

Apparently this leak is a huge boon to SEO. Expect more SERPs to become SEO-overtaken garbage dumps in the near future. Truly depressing.

ad404b8a372f2b9 4y ago
I'm doubtful this will have much of an effect for three reasons: - These are factors, not the underlying algorithms which use them, which means it's not possible to deduce your own ranking from that data. - Many of those are so vague as to be useless. - The most important factors for SEO are those we already know about.
- zexodus 4y ago
  It's possible to use an algorithmic approach (i.e. linear regression) to derive the weights for each of these factors.
  - toto444 4y ago
    All you need to do is scrape the whole internet beforehand.
  - thrwyndx 4y ago
    A lot of these factors depend on internal yandex data, serp clicks especially - good luck getting those outside of yandex prod, and those are the strongest signals.
  - abm53 4y ago
    Given some samples of search results we still don't know: - The X matrix (e.g. the page rank score) for each result. - The y vector, i.e the score for each result. Although we can observe the relative ranking in each sample (would be interested to hear about techniques to cope with this).
  - michaelmior 4y ago
    This is assuming the weights are combined via some linear function. That's not necessarily the case. It's likely that the ranking algorithm is more complex than: calculate features, multiply by weights, add to get ranking. Sure you'll probably still be able to learn something more by playing with the different features, but I doubt you'll get any real meaningful "weights." Especially when how many features are calculated is a black box.

yodsanklai 4y ago

Could someone explain to me how you go from these ranking facts to an actual sorted list of results? and doing it with an acceptable latency. Also, what happened to Google page rank, is is still relevant today?

fsflover 4y ago
I can't answer, but you should be able to find the answer on https://yacy.net https://yacy.net -- free web search. By the way, it can probably tremendously benefit from the leak.
miohtama 4y ago
For an exhaustive answer, you can check from Yandex source code that was posted on another thread on HN https://news.ycombinator.com/item?id=34525936 https://news.ycombinator.com/item?id=34525936
ad404b8a372f2b9 4y ago
"SEOs have already started analyzing Yandex's search ranking factors, which include PageRank and several other link-related factors" PageRank is factor 0.
WinLychee 4y ago
All of the listed "facts" or "features" can be incorporated into an index (e.g. with Lucene) or fed into ML-models (logistic regression, trees, neural nets, etc). Without going into specifics, I've seen ranking treated as a multi-stage algorithmic problem. Initially, you rank results with a lower-quality but low-latency ranker at the first stage (inverted index, tf-idf, knn, etc), and subsequent stages rerank the top-K results with higher-latency ML models outputting a relevance score. I believe progress is currently being made to combine everything into one giant neural model that just ranks everything from the get-go rather than pass in multiple stages.
ValleZ 4y ago
Pages are sorted by page rank, you pick first 10-100k pages and sort using other factors, then take first 10-100 results and sort using most costly factors.

yamrzou 4y ago

The list of factors: https://yandex-explorer.herokuapp.com/search?q=&o=all https://yandex-explorer.herokuapp.com/search?q=&o=all

nicbvs 4y ago

95 - Ukrainian > It is equal to one if the site has a Ukrainian geoist (i.e. 1 - Ukrainian site)

SXX 4y ago
This is completely unrelated to recent events. Before initial invasion in 2014 many Russian IT companies had huge presence in Ukraine and many Yandex services including search had localized versions and own search ranking, etc. Yandex still have versions for Belarus, Kazahstan, other ex-USSR countries and even Turkey. But chances for international expansion was pretty much destroyd by political situation. Also keep in mind Yandex was different company a decade ago. Back then Ilya Segalovich been CTO there (he died of cancer in 2013) and he supported opposition and even participated in street protests in Moscow.

evolextra 4y ago

[flagged]

thekyle 4y ago

It's interesting that they consider the presence of ads and the presence of Yandex ads as separate factors. I wonder if this implies a priority for pages with Yandex ads.

liveoneggs 4y ago
at least we can be sure google would never consider such a ranking
- bitxbitxbitcoin 4y ago
  Really? /S it’s the only reason why I have Adsense.
- dotancohen 4y ago
  That would explain why some content farms rank about e.g. Stack Overflow and Wikipedia with content scraped from those sites.
  - deleted 4y ago
    [deleted]

RugnirViking 4y ago

Interesting ones I have found: 108 MusicQ: The musicality of the request. The results of the sorcerer Anton Konygin. 276 NightQuery: The request is set mainly at night (and accompanying ones for morning, afternoon etc) 316 IsUa: Domain in the .ua zone (ukraine) 747 WordHostWikiSum: The relative popularity of the Word -Host pair, where Word is the word from the Title article on Wikipedia, and the Host is the host that is referred to in this article. 755 NastyContent: Content ugliness factor. (several of the other factors related to port and "adult" content are defined as multiples of this) 892 ShortVideo: A document is a short video (Tiktok, Reels, Shorts) 894 TelegramPost: Document - post in telegram (note I could not find similar for twitter etc, but they do have for tiktok) 1076 More90SecVisitsShare: The share of visits for which the time spent during the day on the host is more than 90 seconds (they track how long users are spending on pages) 1078 RankHackedNovaPhp: Rank of hacked sites 1309 DistanceToAnkara: The distance from the city from where the request was set to Ankara 1603 YellowImgCount: Average yellow images count on host 1614 NewsAgencyRating: Rating of news agency from agencies.json (Yandex.News resource)

inerte 4y ago
Everything in there makes sense imho except the Yellow Image Count. Anyone knows why that would be a factor?
- encoderer 4y ago
  No, but can you explain the distance to Ankara one?
  - inerte 4y ago
    well now you're making me doubt myself, that's a weird one too. My first guess was "they need to know how far one request is from another, and Ankara is a good as any city" So say suppose you're in New Orleans and I am in San Jose, now we can compare our distance, because geographical distance usually equates to cultural differences and perceptions of quality.

g_sch 4y ago

Maybe I'm just a newbie when it comes to building search algorithms, but how on earth would you maintain or test something like this? From experience, I know that search algorithms using far fewer ranking factors than this are deployed in production and treated as terrifying black boxes that no one wants to touch lest they break something in an unexpected way.

thrwyndx 4y ago
Testing is easy with any half decent development setup. You should have some train/eval datasets and monitor metrics on them during training, this is ML 101. And do live A/B experiments for launch candidates. Maintenance sure is hell, lots of sweat and tears. Just glancing over search/formula/webcommon/select_ranking_models.cpp makes me cringe, they must have many dozens if not hundreds of different models in prod by now. Each of them needing maintenance and lots of training data. Work on new ranking factors I suspect must be also highly frustrating: throwing stuff at wall^W catboost black box and seeing if it sticks, and if it doesn't you'd have little idea why and control over it. Imho google's approach (white-boxish interpretable top level ranking formulas) is far superior and maintainable at scale.