4 ms·

Somewhat click-baity. They are not implying runtime / live intervention (which isn’t possible anyways). Top bad results are tweaked. That is part of making the

by harikb 7y ago

Somewhat click-baity. They are not implying runtime / live intervention (which isn’t possible anyways). Top bad results are tweaked. That is part of making the “algorithm”. Who says that is immoral or illegal?

tyingq 7y ago
Not immoral or illegal, but using human raters to seed an algorithm can make the results worse. For example, it could skew to favoring websites where the rater is familiar with the brand. Which might rank mediocre branded content over less well known sites with much higher quality content. That seems to match what I've seen over the last 10 years with Google.
- shadowgovt 7y ago
  Can you give some examples of higher-quality-content sites that Google search is consistently missing?
  - rahidz 7y ago
    Examine.com is one, here's the prior HN thread. Had it not been for that HN thread I would never have known that site even existed, despite having frequently searched for queries where it would have been extremely relevant: https://news.ycombinator.com/item?id=20676755 https://news.ycombinator.com/item?id=20676755
    - shadowgovt 7y ago
      examine.com is the first organic search result I get for "supplemental vitamin d sleep". Maybe the results are hit-or-miss depending on the subtopic (or Google re-tuned the algorithm in the intervening 3 months).
- cinnamonheart 7y ago
  I worked for Leapforce for a while doing this sort of rating work using Google's guidelines. The guidelines were linked elsewhere in this thread and, at a glance, they are the same ones I used: https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf https://static.googleusercontent.com/media/guidelines.raterh... If you're doing something that would skew the results to favour a certain brand, you'd be going against Google's guidelines and your contract would be terminated quickly. I found the review process of my results to be fairly stringent; they were reviewed frequently to ensure I was staying within the guidelines, and I was told even about slight deviatiations.
  - tyingq 7y ago
    I'm not suggesting it would be on purpose. Trust must have some subconscious effect.
- magicalist 7y ago
  > Not immoral or illegal, but using human raters to seed an algorithm can make the results worse ...versus asking an oracle what the "good" results are? It's not clear what you'd propose as an alternative.
  - tyingq 7y ago
    Mitigating controls, expecting the bias. At the moment it's a Frankenstein set up with a rules/algorithm based ranking subsequently shuffled around by several separate ML runs with different aims. The FTC did once unearth a Google attempt to purposefully influence the human ratings: http://graphics.wsj.com/google-ftc-report/ http://graphics.wsj.com/google-ftc-report/ (long read, but worth it). Find this section of the paper: "Unable to get positive reviews from raters when Google demoted comparison shopping sites, Google changed the raters' criteria to try to get positive results."

by harikb 7y ago

tyingq 7y ago

Not immoral or illegal, but using human raters to seed an algorithm can make the results worse. For example, it could skew to favoring websites where the rater is familiar with the brand. Which might rank mediocre branded content over less well known sites with much higher quality content. That seems to match what I've seen over the last 10 years with Google.

shadowgovt 7y ago
Can you give some examples of higher-quality-content sites that Google search is consistently missing?
- rahidz 7y ago
  Examine.com is one, here's the prior HN thread. Had it not been for that HN thread I would never have known that site even existed, despite having frequently searched for queries where it would have been extremely relevant: https://news.ycombinator.com/item?id=20676755 https://news.ycombinator.com/item?id=20676755
  - shadowgovt 7y ago
    examine.com is the first organic search result I get for "supplemental vitamin d sleep". Maybe the results are hit-or-miss depending on the subtopic (or Google re-tuned the algorithm in the intervening 3 months).
cinnamonheart 7y ago
I worked for Leapforce for a while doing this sort of rating work using Google's guidelines. The guidelines were linked elsewhere in this thread and, at a glance, they are the same ones I used: https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf https://static.googleusercontent.com/media/guidelines.raterh... If you're doing something that would skew the results to favour a certain brand, you'd be going against Google's guidelines and your contract would be terminated quickly. I found the review process of my results to be fairly stringent; they were reviewed frequently to ensure I was staying within the guidelines, and I was told even about slight deviatiations.
- tyingq 7y ago
  I'm not suggesting it would be on purpose. Trust must have some subconscious effect.
magicalist 7y ago
> Not immoral or illegal, but using human raters to seed an algorithm can make the results worse ...versus asking an oracle what the "good" results are? It's not clear what you'd propose as an alternative.
- tyingq 7y ago
  Mitigating controls, expecting the bias. At the moment it's a Frankenstein set up with a rules/algorithm based ranking subsequently shuffled around by several separate ML runs with different aims. The FTC did once unearth a Google attempt to purposefully influence the human ratings: http://graphics.wsj.com/google-ftc-report/ http://graphics.wsj.com/google-ftc-report/ (long read, but worth it). Find this section of the paper: "Unable to get positive reviews from raters when Google demoted comparison shopping sites, Google changed the raters' criteria to try to get positive results."