Yeah, the study included tons of variation in result relevance because a big goal was to develop actual relevance information for understanding NDCG as an evalu

Yeah, the study included tons of variation in result relevance because a big goal was to develop actual relevance information for understanding NDCG as an evaluation metric. One of our conclusions was basically that you can’t use relevance measurements as a function of rank position virtually at all unless you can identify when a user is a member of the very tiny subset that actually hunts through results for relevant ones. In some cases this is tipped off by highly specific queries. But in the general case of highly generic or popular queries for the vast majority of users, they will click the first thing more or less regardless of relevance. It’s more like a reflex at that point than a decision. In a very real sense, Google ad placement at the top of list-view search results is a tactic to monetize totally unconscious reflexes. I think in the future it will be considered in the class of behavior as marketing cigarettes to teenagers or something. It’s not a conscious value proposition, but exploitation of cognitive flaws.