4 ms·

I'm referring to the algorithm for calculating the rank of a hit in the search results. That's not PageRank but the ranking algo. If there are hundreds of inf

I'm referring to the algorithm for calculating the rank of a hit in the search results. That's not PageRank but the ranking algo. If there are hundreds of influences, then it seems overcomplicated. Well it needs to be a general algo that works with all types of content, all fields of knowledge, work around all known types of spam, in all languages, while working on HTML quality. Complicated? No, necessary.

palish 18y ago
That's not PageRank but the ranking algo. That's what I'm referring to! Hehe. I feel insane. Am I really failing at communicating? Let me try again: If the ranking algo considers hundreds of influences, then it is an overcomplicated mess!
- ntoshev 18y ago
  Actual ranking at the time of your search does not involve too much computation, because a lot has been precomputed and stored in those Google indexes. In theory, you can view every search as a function of your query and all the web documents out there. This function is partially evaluated for a fixed universe of documents and super optimized. Evaluating hundreds of parameters is common sense for machine learning tasks. If you look at the papers explaining how the top Netflix prize competitors achieve their score, this is again what they do: they run literally hundreds of algorithms over the dataset and then blend the results. And this is for a domain where their data is only (user, movie, rating, date) quadriplets.
  - palish 18y ago
    Yeah... But just because a design is also a scientific paper doesn't make it overcomplicated.
    - ntoshev 18y ago
      Don't get your point... Mine was that the solutions to machine learning problems are inherently messy (at least this is the state of the art).
      - palish 18y ago
        Typo, sorry. I meant, just because a design is a scientific paper, that doesn't make it a good or simple one.
- Mystalic 18y ago
  Google keeps it very clean, and they tune the algo every week. But again, it's necessary because relevancy to a human being is continent on THOUSANDS of factors, and they try to incorporate as much of that in as simple of an algorithm as possible.
- anamax 18y ago
  > If the ranking algo considers hundreds of influences, then it is an overcomplicated mess! "overcomplicated mess" seems to imply "it's a bad solution". While Google's ranking system may be "bad" (they believe that it isn't good enough), "bad" or even "overcomplicated" doesn't follow from "hundreds of influences". Consider the "design" of a common house cat. Its design is the result of thousands of influences. It may be difficult for people to think about systems with hundreds of influences, but that's a people problem and doesn't imply that such systems are inferior to more understandable systems.
  - palish 18y ago
    Housecats weren't designed, they were evolved. That's a big difference. There can be cleanliness and simplicity in design. Also, a housecat's goal is to live and to reproduce, which involves solving many problems, whereas the ranking algorithm's goal is to match text to relevant pages, which requires solving far fewer problems than a living creature, so it's really not a fair metaphor.
    - anamax 18y ago
      > Housecats weren't designed, they were evolved. That's a big difference. What properties of a design do you think are determined by the design process as opposed to its outcome? > whereas the ranking algorithm's goal is to match text to relevant pages, which requires solving far fewer problems than a living creature While "relevance" is the commonly used term, "utility" or "value" would be more accurate. And they're far more complicated than you seem to think. There is no "best" search result for a given query. There's not even a best result for a given person. Add in their context, and you're getting close, but there are still other factors that are relevant to determining "best".
    - bd 18y ago
      I suspect a large part of ranking algorithm's complexity comes also from evolution - they need to coevolve with all SEO stuff. Optimal ranking is a moving target, parasites break your fitness.

palish 18y ago

That's not PageRank but the ranking algo. That's what I'm referring to! Hehe. I feel insane. Am I really failing at communicating? Let me try again: If the ranking algo considers hundreds of influences, then it is an overcomplicated mess!

ntoshev 18y ago
Actual ranking at the time of your search does not involve too much computation, because a lot has been precomputed and stored in those Google indexes. In theory, you can view every search as a function of your query and all the web documents out there. This function is partially evaluated for a fixed universe of documents and super optimized. Evaluating hundreds of parameters is common sense for machine learning tasks. If you look at the papers explaining how the top Netflix prize competitors achieve their score, this is again what they do: they run literally hundreds of algorithms over the dataset and then blend the results. And this is for a domain where their data is only (user, movie, rating, date) quadriplets.
- palish 18y ago
  Yeah... But just because a design is also a scientific paper doesn't make it overcomplicated.
  - ntoshev 18y ago
    Don't get your point... Mine was that the solutions to machine learning problems are inherently messy (at least this is the state of the art).
    - palish 18y ago
      Typo, sorry. I meant, just because a design is a scientific paper, that doesn't make it a good or simple one.
Mystalic 18y ago
Google keeps it very clean, and they tune the algo every week. But again, it's necessary because relevancy to a human being is continent on THOUSANDS of factors, and they try to incorporate as much of that in as simple of an algorithm as possible.
anamax 18y ago
> If the ranking algo considers hundreds of influences, then it is an overcomplicated mess! "overcomplicated mess" seems to imply "it's a bad solution". While Google's ranking system may be "bad" (they believe that it isn't good enough), "bad" or even "overcomplicated" doesn't follow from "hundreds of influences". Consider the "design" of a common house cat. Its design is the result of thousands of influences. It may be difficult for people to think about systems with hundreds of influences, but that's a people problem and doesn't imply that such systems are inferior to more understandable systems.
- palish 18y ago
  Housecats weren't designed, they were evolved. That's a big difference. There can be cleanliness and simplicity in design. Also, a housecat's goal is to live and to reproduce, which involves solving many problems, whereas the ranking algorithm's goal is to match text to relevant pages, which requires solving far fewer problems than a living creature, so it's really not a fair metaphor.
  - anamax 18y ago
    > Housecats weren't designed, they were evolved. That's a big difference. What properties of a design do you think are determined by the design process as opposed to its outcome? > whereas the ranking algorithm's goal is to match text to relevant pages, which requires solving far fewer problems than a living creature While "relevance" is the commonly used term, "utility" or "value" would be more accurate. And they're far more complicated than you seem to think. There is no "best" search result for a given query. There's not even a best result for a given person. Add in their context, and you're getting close, but there are still other factors that are relevant to determining "best".
  - bd 18y ago
    I suspect a large part of ranking algorithm's complexity comes also from evolution - they need to coevolve with all SEO stuff. Optimal ranking is a moving target, parasites break your fitness.