4 ms·

Could someone explain to me how you go from these ranking facts to an actual sorted list of results? and doing it with an acceptable latency. Also, what happene

Could someone explain to me how you go from these ranking facts to an actual sorted list of results? and doing it with an acceptable latency. Also, what happened to Google page rank, is is still relevant today?

fsflover 4y ago
I can't answer, but you should be able to find the answer on https://yacy.net https://yacy.net -- free web search. By the way, it can probably tremendously benefit from the leak.
miohtama 4y ago
For an exhaustive answer, you can check from Yandex source code that was posted on another thread on HN https://news.ycombinator.com/item?id=34525936 https://news.ycombinator.com/item?id=34525936
ad404b8a372f2b9 4y ago
"SEOs have already started analyzing Yandex's search ranking factors, which include PageRank and several other link-related factors" PageRank is factor 0.
WinLychee 4y ago
All of the listed "facts" or "features" can be incorporated into an index (e.g. with Lucene) or fed into ML-models (logistic regression, trees, neural nets, etc). Without going into specifics, I've seen ranking treated as a multi-stage algorithmic problem. Initially, you rank results with a lower-quality but low-latency ranker at the first stage (inverted index, tf-idf, knn, etc), and subsequent stages rerank the top-K results with higher-latency ML models outputting a relevance score. I believe progress is currently being made to combine everything into one giant neural model that just ranks everything from the get-go rather than pass in multiple stages.
ValleZ 4y ago
Pages are sorted by page rank, you pick first 10-100k pages and sort using other factors, then take first 10-100 results and sort using most costly factors.
thrwyndx 4y ago
You throw them into a machine learning model together with a big dataset of queries/urls annotated by humans for relevancy. Catboost is yandex's choice of model here. > and doing it with an acceptable latency Lots of interesting optimizations possible here, but the big obvious one is multiple level models: score documents with a cheap model (FastRank in yandex lingvo) first using a subset of the fastest available features, then rescore top docs with your best slow expensive model. Perhaps rescore multiple times at different points in the stack with models of varying complexity, at each index shard and after aggregating the results from subset/all shards. Also sort documents in each index shard by some other ML model with query-independent features to push all the junk to the end of the index where you'd likely skip it when running out of time budget to process a query. > Also, what happened to Google page rank, is is still relevant today? Vanilla 1990s' pagerank obviously not, but the idea of such graph-based calculations is still very useful yes.
- swyx 4y ago
  > Vanilla 1990s' pagerank obviously not, what did we learn about the flaws?
  - thrwyndx 4y ago
    It's old and too simple for today's web, everyone knows it and everyone games it. But the idea behind it is still useful, just need more tricks, more ML etc.

fsflover 4y ago

I can't answer, but you should be able to find the answer on https://yacy.net https://yacy.net -- free web search. By the way, it can probably tremendously benefit from the leak.

miohtama 4y ago

For an exhaustive answer, you can check from Yandex source code that was posted on another thread on HN https://news.ycombinator.com/item?id=34525936 https://news.ycombinator.com/item?id=34525936

ad404b8a372f2b9 4y ago

"SEOs have already started analyzing Yandex's search ranking factors, which include PageRank and several other link-related factors" PageRank is factor 0.

WinLychee 4y ago

All of the listed "facts" or "features" can be incorporated into an index (e.g. with Lucene) or fed into ML-models (logistic regression, trees, neural nets, etc). Without going into specifics, I've seen ranking treated as a multi-stage algorithmic problem. Initially, you rank results with a lower-quality but low-latency ranker at the first stage (inverted index, tf-idf, knn, etc), and subsequent stages rerank the top-K results with higher-latency ML models outputting a relevance score. I believe progress is currently being made to combine everything into one giant neural model that just ranks everything from the get-go rather than pass in multiple stages.

ValleZ 4y ago

Pages are sorted by page rank, you pick first 10-100k pages and sort using other factors, then take first 10-100 results and sort using most costly factors.

thrwyndx 4y ago

You throw them into a machine learning model together with a big dataset of queries/urls annotated by humans for relevancy. Catboost is yandex's choice of model here. > and doing it with an acceptable latency Lots of interesting optimizations possible here, but the big obvious one is multiple level models: score documents with a cheap model (FastRank in yandex lingvo) first using a subset of the fastest available features, then rescore top docs with your best slow expensive model. Perhaps rescore multiple times at different points in the stack with models of varying complexity, at each index shard and after aggregating the results from subset/all shards. Also sort documents in each index shard by some other ML model with query-independent features to push all the junk to the end of the index where you'd likely skip it when running out of time budget to process a query. > Also, what happened to Google page rank, is is still relevant today? Vanilla 1990s' pagerank obviously not, but the idea of such graph-based calculations is still very useful yes.

swyx 4y ago
> Vanilla 1990s' pagerank obviously not, what did we learn about the flaws?
- thrwyndx 4y ago
  It's old and too simple for today's web, everyone knows it and everyone games it. But the idea behind it is still useful, just need more tricks, more ML etc.