3 ms·

I work in ML for search engines and have been part of very large cohort (n > 5000) user studies carried out by a former employer to measure eye saccade movement

by mlthoughts2018 7y ago

I work in ML for search engines and have been part of very large cohort (n > 5000) user studies carried out by a former employer to measure eye saccade movements over search result pages to understand where user attention is drawn in different search modalities. In list form results, users typically just click whatever is the first thing without much consideration for its relevance. In gallery form results, users will scan in a ”backwards C” shape along the first row, down the far right column, then from right to left along the bottom row above the fold, pretty consistently, but still usually just click things in the first row. The cohort was meticulously designed to allow controlling for factors like age, search interface familiarity (users completed tasks in sessions prior to the eye measurement portion), country, language, etc.

ijpoijpoihpiuoh 7y ago
Interesting! It might go without saying, but just to check: I assume the study intentionally included some cases where the first result was not the most relevant one, right? Did the study find differences in the rate of clicking on the first result when the first result was irrelevant?
- mlthoughts2018 7y ago
  Yeah, the study included tons of variation in result relevance because a big goal was to develop actual relevance information for understanding NDCG as an evaluation metric. One of our conclusions was basically that you can’t use relevance measurements as a function of rank position virtually at all unless you can identify when a user is a member of the very tiny subset that actually hunts through results for relevant ones. In some cases this is tipped off by highly specific queries. But in the general case of highly generic or popular queries for the vast majority of users, they will click the first thing more or less regardless of relevance. It’s more like a reflex at that point than a decision. In a very real sense, Google ad placement at the top of list-view search results is a tactic to monetize totally unconscious reflexes. I think in the future it will be considered in the class of behavior as marketing cigarettes to teenagers or something. It’s not a conscious value proposition, but exploitation of cognitive flaws.
eclarkso 7y ago
There is a much older, smaller - but public - study done by MSR a long while ago as well: https://www.microsoft.com/en-us/research/publication/what-are-you-looking-for-an-eye-tracking-study-of-information-usage-in-web-search/ https://www.microsoft.com/en-us/research/publication/what-ar...

ijpoijpoihpiuoh 7y ago

Interesting! It might go without saying, but just to check: I assume the study intentionally included some cases where the first result was not the most relevant one, right? Did the study find differences in the rate of clicking on the first result when the first result was irrelevant?

mlthoughts2018 7y ago
Yeah, the study included tons of variation in result relevance because a big goal was to develop actual relevance information for understanding NDCG as an evaluation metric. One of our conclusions was basically that you can’t use relevance measurements as a function of rank position virtually at all unless you can identify when a user is a member of the very tiny subset that actually hunts through results for relevant ones. In some cases this is tipped off by highly specific queries. But in the general case of highly generic or popular queries for the vast majority of users, they will click the first thing more or less regardless of relevance. It’s more like a reflex at that point than a decision. In a very real sense, Google ad placement at the top of list-view search results is a tactic to monetize totally unconscious reflexes. I think in the future it will be considered in the class of behavior as marketing cigarettes to teenagers or something. It’s not a conscious value proposition, but exploitation of cognitive flaws.

eclarkso 7y ago

There is a much older, smaller - but public - study done by MSR a long while ago as well: https://www.microsoft.com/en-us/research/publication/what-are-you-looking-for-an-eye-tracking-study-of-information-usage-in-web-search/ https://www.microsoft.com/en-us/research/publication/what-ar...