5 ms·

> I used to work at Bing. If you really want Bing to improve, the best thing you can do is just use it: clicks on search results, plus backs and dwell times, ar

by clarry 7y ago

> I used to work at Bing. If you really want Bing to improve, the best thing you can do is just use it: clicks on search results, plus backs and dwell times, are vital training data. A million flies are attracted to shit. I don't believe this sort of training data will ever become useful if it's in the same pool with the rest of the world. See also: voting with your wallet against the tyranny of the majority of uninformed consumers who buy whatever is most marketed. Those pennies don't matter. In fact, I believe this sort of training and optimization for the mainstream plays a role in allowing bad results to proliferate. This is before we even consider the fact that clicking on many results can indicate that they're bad (I click another result because the previous wasn't good), or because they're good (I'm browsing choices). Dwelling long can be bad (crappy & slow site, it takes me long to find the information I want or turn away) or it can be good (I found good stuff and I'm spending a while on it). Whatever conclusion your training system draws might be completely wrong. And probably prone to being gamed.

mturmon 7y ago
The above heuristic arguments seem inherently weaker than the direct experience of someone who used to be on the team that improves the results. They are well-funded and should be able to back out the effects you mention.
- clarry 7y ago
  How are they ever going to know whether they improved results for me or not? You might as well train AI to play a game without ever checking their score. Oh, it's spending 20% more time in each room now and firing fewer bullets than before. Surely it is a better AI now. Sorry, appeal to authority is no argument. Direct experience is valuable if there's an argument or some real scenario we can dissect, otherwise it's nothing more than a baseless claim. Without concrete examples, it's not even an anecdote. And there are plenty of anecdotes about search results for in-depth content becoming harder and harder to find. The person I was responding to posted that they're using this training data to order the top 10 results or so. That's already an indication that it's not very helpful for me. I don't get frustrated if the top few results are in suboptimal order, I get frustrated when I get pages and pages of garbage and irrelevant results and can't seem to get anything useful out of it.
JohnBooty 7y ago
See the replies elsewhere. The metrics go beyond simple, naive "click counting." Specifically they also measure metrics like "regret" - if you went back to the search results and chose something else this indicates it wasn't a useful search result for the query and this counts against it. In theory this is quite a nice defense against clickbaity, content free results.
- clarry 7y ago
  > Specifically they also measure metrics like "regret" I explicitly addressed this: > clicking on many results can indicate that they're bad (I click another result because the previous wasn't good), or because they're good (I'm browsing choices). You don't know the reason why I clicked back. Maybe I got what I needed at a glance, and I'm browsing to the next thing to see if there's more of what I need. You don't know that the Average Joe (and a million other Average Joes) wasted 15 minutes on clickbaity content free crap despite not getting anything useful out of it. You know, those same SEO spam sites are trying to find ways to make the user stay as long as possible; they are in the game. And I'm sure there are plenty of gullible users who will stay. Who knows, maybe SEO spam is working and that's precisely why we see SEO spam sites in results? Again, I think training like this is pretending the problem is much simpler than it is. And I think that there is a real danger that training like this just optimizes for 1) average users 2) sites that managed to game the system with their SEO. Neither is optimizing for quality results. That theory has so many holes and pitfalls.
  - neodymiumphish 7y ago
    I just think you're assuming these search engines are using way fewer factors than they actually use. It's probably exceptionally easy to tie your search behavior to someone else who's much more like you than "the average", unless you're using a search engine like DDG, Qwant, etc. Maybe the best bet is for a search engine to try to include a simple, non-intrusive method of getting user input to determine the usefulness of a search result. Like, if it determines you came back as a "regret", maybe a ThumbsUp/ThumbsDown icon next to the last link you clicked to get you to say whether that link was useful in any way, or maybe a "block" option as well to say that you never want sites like that again.
neodymiumphish 7y ago
A lot of this does, in fact, get solved through machine learning. If one search query is conducted a hundred times by different people with otherwise similar profiles, the machine learning can constantly modify the results until the hundredth person sees the best results first. For example, if the first 25 people all click on the 2nd, 4th, 5th, and 8th result and then stop, it can modify the order of the results for the next 25 people, and randomize the order, so that some see 8-4-2-5 first, and others see 2-5-8-4, etc. The it re-assesses the results and gets smarter. The gaming part is where there's a lot of importance, since it forces the machine to learn which users are acting nefariously.
- clarry 7y ago
  Shuffling the top 10 results is not fixing the big problem in search for me. I don't care too much whether the company I search for is in first or fourth place. But I'm really irritated when I get pages upon pages of useless results and nothing remotely relevant.
pheug 7y ago
Clicks are noisy and have their problems (especially clickbait), but as far as ranking goes they still blow out of water everything else you can use. Analemma_ is right on the money. We need more Lemon Pledge^W^W clicks! Speaking as someone who worked for another big web search engine in the past (not G, not B) And counting good/bad clicks for a given (query, url) pair is just the tip of the iceberg. There's a lot of other interesting stuff you can do with them _if_ you have the data. Deep learning/NLP with clicks as training signal is probably the most exciting area to me, to name one. Unfortunately almost all of the data currently gets nabbed by google, other search engines are just getting by with the scraps. And it's very hard to bootstrap a competitor from scratch - for example Cliqz had to resort to some shady deals with Mozilla just to get any data to start with.

by clarry 7y ago

mturmon 7y ago

The above heuristic arguments seem inherently weaker than the direct experience of someone who used to be on the team that improves the results. They are well-funded and should be able to back out the effects you mention.

clarry 7y ago
How are they ever going to know whether they improved results for me or not? You might as well train AI to play a game without ever checking their score. Oh, it's spending 20% more time in each room now and firing fewer bullets than before. Surely it is a better AI now. Sorry, appeal to authority is no argument. Direct experience is valuable if there's an argument or some real scenario we can dissect, otherwise it's nothing more than a baseless claim. Without concrete examples, it's not even an anecdote. And there are plenty of anecdotes about search results for in-depth content becoming harder and harder to find. The person I was responding to posted that they're using this training data to order the top 10 results or so. That's already an indication that it's not very helpful for me. I don't get frustrated if the top few results are in suboptimal order, I get frustrated when I get pages and pages of garbage and irrelevant results and can't seem to get anything useful out of it.

JohnBooty 7y ago

See the replies elsewhere. The metrics go beyond simple, naive "click counting." Specifically they also measure metrics like "regret" - if you went back to the search results and chose something else this indicates it wasn't a useful search result for the query and this counts against it. In theory this is quite a nice defense against clickbaity, content free results.

clarry 7y ago
> Specifically they also measure metrics like "regret" I explicitly addressed this: > clicking on many results can indicate that they're bad (I click another result because the previous wasn't good), or because they're good (I'm browsing choices). You don't know the reason why I clicked back. Maybe I got what I needed at a glance, and I'm browsing to the next thing to see if there's more of what I need. You don't know that the Average Joe (and a million other Average Joes) wasted 15 minutes on clickbaity content free crap despite not getting anything useful out of it. You know, those same SEO spam sites are trying to find ways to make the user stay as long as possible; they are in the game. And I'm sure there are plenty of gullible users who will stay. Who knows, maybe SEO spam is working and that's precisely why we see SEO spam sites in results? Again, I think training like this is pretending the problem is much simpler than it is. And I think that there is a real danger that training like this just optimizes for 1) average users 2) sites that managed to game the system with their SEO. Neither is optimizing for quality results. That theory has so many holes and pitfalls.
- neodymiumphish 7y ago
  I just think you're assuming these search engines are using way fewer factors than they actually use. It's probably exceptionally easy to tie your search behavior to someone else who's much more like you than "the average", unless you're using a search engine like DDG, Qwant, etc. Maybe the best bet is for a search engine to try to include a simple, non-intrusive method of getting user input to determine the usefulness of a search result. Like, if it determines you came back as a "regret", maybe a ThumbsUp/ThumbsDown icon next to the last link you clicked to get you to say whether that link was useful in any way, or maybe a "block" option as well to say that you never want sites like that again.

neodymiumphish 7y ago

A lot of this does, in fact, get solved through machine learning. If one search query is conducted a hundred times by different people with otherwise similar profiles, the machine learning can constantly modify the results until the hundredth person sees the best results first. For example, if the first 25 people all click on the 2nd, 4th, 5th, and 8th result and then stop, it can modify the order of the results for the next 25 people, and randomize the order, so that some see 8-4-2-5 first, and others see 2-5-8-4, etc. The it re-assesses the results and gets smarter. The gaming part is where there's a lot of importance, since it forces the machine to learn which users are acting nefariously.

clarry 7y ago
Shuffling the top 10 results is not fixing the big problem in search for me. I don't care too much whether the company I search for is in first or fourth place. But I'm really irritated when I get pages upon pages of useless results and nothing remotely relevant.

pheug 7y ago

Clicks are noisy and have their problems (especially clickbait), but as far as ranking goes they still blow out of water everything else you can use. Analemma_ is right on the money. We need more Lemon Pledge^W^W clicks! Speaking as someone who worked for another big web search engine in the past (not G, not B) And counting good/bad clicks for a given (query, url) pair is just the tip of the iceberg. There's a lot of other interesting stuff you can do with them _if_ you have the data. Deep learning/NLP with clicks as training signal is probably the most exciting area to me, to name one. Unfortunately almost all of the data currently gets nabbed by google, other search engines are just getting by with the scraps. And it's very hard to bootstrap a competitor from scratch - for example Cliqz had to resort to some shady deals with Mozilla just to get any data to start with.