4 ms·

How many people have you personally observed searching for Basecamp? I mean, it can't be that many, right? Or, do you mean, most people you've watched use Googl

How many people have you personally observed searching for Basecamp? I mean, it can't be that many, right? Or, do you mean, most people you've watched use Google just click the top link for whatever they're searching for? But even then, how often do you look over people's shoulders when they're searching? Maybe I'm weird but, like, I don't have any idea how my friends or colleagues use Google, and I work in the industry. So I'm really curious via what mechanism you were able to gather enough samples that you feel you can make a strong statement on this one way or another. Are you in SEO? A UX researcher? Do you have a business interest in the way people use Google, somehow?

mlthoughts2018 7y ago
I work in ML for search engines and have been part of very large cohort (n > 5000) user studies carried out by a former employer to measure eye saccade movements over search result pages to understand where user attention is drawn in different search modalities. In list form results, users typically just click whatever is the first thing without much consideration for its relevance. In gallery form results, users will scan in a ”backwards C” shape along the first row, down the far right column, then from right to left along the bottom row above the fold, pretty consistently, but still usually just click things in the first row. The cohort was meticulously designed to allow controlling for factors like age, search interface familiarity (users completed tasks in sessions prior to the eye measurement portion), country, language, etc.
- ijpoijpoihpiuoh 7y ago
  Interesting! It might go without saying, but just to check: I assume the study intentionally included some cases where the first result was not the most relevant one, right? Did the study find differences in the rate of clicking on the first result when the first result was irrelevant?
  - mlthoughts2018 7y ago
    Yeah, the study included tons of variation in result relevance because a big goal was to develop actual relevance information for understanding NDCG as an evaluation metric. One of our conclusions was basically that you can’t use relevance measurements as a function of rank position virtually at all unless you can identify when a user is a member of the very tiny subset that actually hunts through results for relevant ones. In some cases this is tipped off by highly specific queries. But in the general case of highly generic or popular queries for the vast majority of users, they will click the first thing more or less regardless of relevance. It’s more like a reflex at that point than a decision. In a very real sense, Google ad placement at the top of list-view search results is a tactic to monetize totally unconscious reflexes. I think in the future it will be considered in the class of behavior as marketing cigarettes to teenagers or something. It’s not a conscious value proposition, but exploitation of cognitive flaws.
- eclarkso 7y ago
  There is a much older, smaller - but public - study done by MSR a long while ago as well: https://www.microsoft.com/en-us/research/publication/what-are-you-looking-for-an-eye-tracking-study-of-information-usage-in-web-search/ https://www.microsoft.com/en-us/research/publication/what-ar...
svachalek 7y ago
There's a great answer here on search engine studies but I want to point out the position the parent comment is making: most people don't know the difference between using a search engine like Google and using the WWW itself. Domain names, URLs, protocols, search engines, links, paid placement, they're all just annoying babble between you and the cat pictures you're looking for. Fortunately, you can learn the clicks and the typing you need after a few tries and then it's just muscle memory to repeat at will; you never have to figure all that stuff out. If you work in "the industry" you owe it to yourself to watch "real users" for a while. Sure we're talking about anecdotes here but you should have your own.

mlthoughts2018 7y ago

I work in ML for search engines and have been part of very large cohort (n > 5000) user studies carried out by a former employer to measure eye saccade movements over search result pages to understand where user attention is drawn in different search modalities. In list form results, users typically just click whatever is the first thing without much consideration for its relevance. In gallery form results, users will scan in a ”backwards C” shape along the first row, down the far right column, then from right to left along the bottom row above the fold, pretty consistently, but still usually just click things in the first row. The cohort was meticulously designed to allow controlling for factors like age, search interface familiarity (users completed tasks in sessions prior to the eye measurement portion), country, language, etc.

ijpoijpoihpiuoh 7y ago
Interesting! It might go without saying, but just to check: I assume the study intentionally included some cases where the first result was not the most relevant one, right? Did the study find differences in the rate of clicking on the first result when the first result was irrelevant?
- mlthoughts2018 7y ago
  Yeah, the study included tons of variation in result relevance because a big goal was to develop actual relevance information for understanding NDCG as an evaluation metric. One of our conclusions was basically that you can’t use relevance measurements as a function of rank position virtually at all unless you can identify when a user is a member of the very tiny subset that actually hunts through results for relevant ones. In some cases this is tipped off by highly specific queries. But in the general case of highly generic or popular queries for the vast majority of users, they will click the first thing more or less regardless of relevance. It’s more like a reflex at that point than a decision. In a very real sense, Google ad placement at the top of list-view search results is a tactic to monetize totally unconscious reflexes. I think in the future it will be considered in the class of behavior as marketing cigarettes to teenagers or something. It’s not a conscious value proposition, but exploitation of cognitive flaws.
eclarkso 7y ago
There is a much older, smaller - but public - study done by MSR a long while ago as well: https://www.microsoft.com/en-us/research/publication/what-are-you-looking-for-an-eye-tracking-study-of-information-usage-in-web-search/ https://www.microsoft.com/en-us/research/publication/what-ar...

svachalek 7y ago

There's a great answer here on search engine studies but I want to point out the position the parent comment is making: most people don't know the difference between using a search engine like Google and using the WWW itself. Domain names, URLs, protocols, search engines, links, paid placement, they're all just annoying babble between you and the cat pictures you're looking for. Fortunately, you can learn the clicks and the typing you need after a few tries and then it's just muscle memory to repeat at will; you never have to figure all that stuff out. If you work in "the industry" you owe it to yourself to watch "real users" for a while. Sure we're talking about anecdotes here but you should have your own.