3 ms·

Yup. This is the Long Click metric. Evaluating search is difficult because it's a tension: if users click a lot, is it because they find many valuable things,

by kqr 3y ago

Yup. This is the Long Click metric. Evaluating search is difficult because it's a tension: if users click a lot, is it because they find many valuable things, or because they didn't find what they were looking for? If a user clicked just once, is it because they found what they were looking for or just that the rest of the results were so bad the user gave up? The long click (user clicked, then didn't click again for a while) is a better metric, but also not ideal: did they stay because they found what they were looking for, or was the result just that confusing they had to stay to comprehend whether it was the right thing? Most often it's because they found what they were looking for, but the pathological cases hide in the middle: many similar correct results, winner is the one that makes the user a little slower. (This has nothing to do with tabs or back buttons, by the way. It happens any time they can detect subsequent clicks on the search result page.) I've worked in the search space (though on less evil projects than Google) and I still struggle with the question on how to evaluate search. If you have ideas, let me know!

plank 3y ago
One idea, but people will probably hate me for it: If you return to e.g. the google search site (hence: when the long click metric would be triggered) have a dialog on top saying ‘result great / OK / bad-or-confusing’. Can probably be gamed (bot nets trying to destroy the reputation of others) but at least a long time would not automatically mean ‘great result’. (In the arms race to combat destruction, it could be so that a ‘bad-or-confusing’ click would not actually push a value down, just not make it go higher). Kind regards, Roel
- squidbeak 3y ago
  This was tried with a +1 button around the time of Google Plus's launch.
chillfox 3y ago
I feel like the problem is trying to turn human experience into a metric. Probably the better approach would be to have a well staffed QA team.
- makeitdouble 3y ago
  We should be mad at Yahoo for having fucked up. If anything, they could have spun out the search part and be remembered for it,.
- agileAlligator 3y ago
  I honestly don't think it's possible to have a QA team large enough to handle the gajillions of websites that come up and disappear every day. They just have to come up with better and better metrics until they find one that approximates the human experience the best.
  - sojournerc 3y ago
    You don't have to cover the long tail... Maybe just top 10% of topics would be a big improvement.
withinboredom 3y ago
> if users click a lot, is it because they find many valuable things, or because they didn't find what they were looking for? Why do you care as a search engine? This is a natural human problem that can't be solved with technology, only by humans. It used to be, that I went to page 5 of Google instantly, because that was where the real results were. The first few pages were people who knew more SEO than sense. These days, that doesn't work since "semantic search" because now it appears to be sorted by some relevance metric and by about page 5 you start getting into "marginally related to some definition of what you typed in but still knows too much SEO to be useful." The point is, this was already a solved problem if you knew to go to about page 4-5. Then people started trying to use a technical solution to a very human problem.
- kqr 3y ago
  > Why do you care as a search engine? Wait, are you really asking why a search engine would care how well it finds what the user is looking for? Granted, there are a lot of search engines that sell themselves on other metrics ("it's fast!" or "it uses AI!" or "it's in the cloud!") but any serious search engine player strives to learn how good it is -- in practise -- at helping the user find what they are looking for. That's ultimately the purpose of a search engine.
  - withinboredom 3y ago
    > Wait, are you really asking why a search engine would care how well it finds what the user is looking for? While a useful metric, it's an unknowable metric. 1. You have no idea if the user even knows what they are looking for, so how would you know that they found it? 2. You have no idea if the user found what they are looking for, maybe what they are looking for isn't on the internet? 3. You have no idea if the user is even looking for something, maybe it was just a cat running across the keyboard? The only way to learn the answer is to have humans talk to humans. You can't game your way through it by using metrics. It reminds me of this one time the CEO asked our team to add a metric for "successful websites" (we were a hosting provider) and we rebuffed with "define successful." They immediately mentioned page views, which we replied "what about a restaurant with a downloadable menu that google links to directly?" and back and forth with "successful" never being defined for all verticals and all cases. It just isn't possible to define using heuristics.

by kqr 3y ago

plank 3y ago

One idea, but people will probably hate me for it: If you return to e.g. the google search site (hence: when the long click metric would be triggered) have a dialog on top saying ‘result great / OK / bad-or-confusing’. Can probably be gamed (bot nets trying to destroy the reputation of others) but at least a long time would not automatically mean ‘great result’. (In the arms race to combat destruction, it could be so that a ‘bad-or-confusing’ click would not actually push a value down, just not make it go higher). Kind regards, Roel

squidbeak 3y ago
This was tried with a +1 button around the time of Google Plus's launch.

chillfox 3y ago

I feel like the problem is trying to turn human experience into a metric. Probably the better approach would be to have a well staffed QA team.

makeitdouble 3y ago
We should be mad at Yahoo for having fucked up. If anything, they could have spun out the search part and be remembered for it,.
agileAlligator 3y ago
I honestly don't think it's possible to have a QA team large enough to handle the gajillions of websites that come up and disappear every day. They just have to come up with better and better metrics until they find one that approximates the human experience the best.
- sojournerc 3y ago
  You don't have to cover the long tail... Maybe just top 10% of topics would be a big improvement.

withinboredom 3y ago

> if users click a lot, is it because they find many valuable things, or because they didn't find what they were looking for? Why do you care as a search engine? This is a natural human problem that can't be solved with technology, only by humans. It used to be, that I went to page 5 of Google instantly, because that was where the real results were. The first few pages were people who knew more SEO than sense. These days, that doesn't work since "semantic search" because now it appears to be sorted by some relevance metric and by about page 5 you start getting into "marginally related to some definition of what you typed in but still knows too much SEO to be useful." The point is, this was already a solved problem if you knew to go to about page 4-5. Then people started trying to use a technical solution to a very human problem.

kqr 3y ago
> Why do you care as a search engine? Wait, are you really asking why a search engine would care how well it finds what the user is looking for? Granted, there are a lot of search engines that sell themselves on other metrics ("it's fast!" or "it uses AI!" or "it's in the cloud!") but any serious search engine player strives to learn how good it is -- in practise -- at helping the user find what they are looking for. That's ultimately the purpose of a search engine.
- withinboredom 3y ago
  > Wait, are you really asking why a search engine would care how well it finds what the user is looking for? While a useful metric, it's an unknowable metric. 1. You have no idea if the user even knows what they are looking for, so how would you know that they found it? 2. You have no idea if the user found what they are looking for, maybe what they are looking for isn't on the internet? 3. You have no idea if the user is even looking for something, maybe it was just a cat running across the keyboard? The only way to learn the answer is to have humans talk to humans. You can't game your way through it by using metrics. It reminds me of this one time the CEO asked our team to add a metric for "successful websites" (we were a hosting provider) and we rebuffed with "define successful." They immediately mentioned page views, which we replied "what about a restaurant with a downloadable menu that google links to directly?" and back and forth with "successful" never being defined for all verticals and all cases. It just isn't possible to define using heuristics.