3 ms·

Maybe I'm reading it differently than you, but I don't see that statement as being particularly meaningful. All that says is that they didn't have access to Goo

by eridius 11y ago

Maybe I'm reading it differently than you, but I don't see that statement as being particularly meaningful. All that says is that they didn't have access to Google's actual internal data on clickthrough rates. Which is not surprising. It turns out what they did was use a site called UsabilityHub (which is intended for testing website designs) to show the study participants a screenshot of search results. One screenshot showed Google's results, while another shows search results that were purportedly sorted purely by relevance. I'm assuming users simply reported whether they would have clicked on any results on the page, but the article doesn't go into that level of detail. I also assume the users weren't told whether they were looking at the genuine results or not. Given that, there's nothing at all surprising about that statement you highlighted. In fact, unless Google were willing to actually run the study themselves, randomly showing either their current results or results sorted purely by "relevance", and gathering the real clickthrough rates, then what the article describes here seems like the only real reasonable way to do this.

downandout 11y ago
One screenshot showed Google's results, while another shows search results that were purportedly sorted purely by relevance. Therein lies the problem. They don't know how Google would order the results in the hypothetical scenario where results didn't include those that make Google money. This goes beyond not having clickthrough data; what they were showing to their test users was simply a not-very-good, human interpretation of what they presume the Google algorithm might return in a nonexistent situation. Unsurprisingly, these fictional search results had worse CTR's than were achieved by displaying presumably custom-crafted Yelp results. The term relevance is unique to each search algorithm, and these researchers did not have the data to determine what the relevance would otherwise be for Google. This report essentially says "if we ran Google, we could get better CTR's by showing Yelp results". That does not prove consumer harm.
- eridius 11y ago
  > They don't know how Google would order the results in the hypothetical scenario where results didn't include those that make Google money. They kind of do. Google's very public about most of how they sort results, and the article said they used Google's "algorithm" to rank them. I know Google used to even expose the underlying PageRank number for various results, but I don't know if they still do so today (if so, then it becomes even easier to sort them the way you think Google would). > what they were showing to their test users was simply a not-very-good, human interpretation of what they presume the Google algorithm might return in a nonexistent situation You're making a very bad assumption here. From the article: > One set of users saw a page reflecting results currently displayed by Google Based on that it sure sounds like they actually performed various searches on Google, screenshotted the results, then fabricated a second set of results that reflects what they believed Google would have shown if it sorted purely by "relevance" instead of prioritizing its own services. And these fabricated results had significantly higher CTRs than the genuine results from Google. > The term relevance is unique to each search algorithm, and these researchers did not have the data to determine what the relevance would otherwise be for Google. Sure they do. The way Google ranks search results is quite public and has been so from the beginning. What isn't public is all the various tweaks they perform to the results, such as removing what they believe to be spam or content farms, as well as the biasing they do in favor of their own services. But the core algorithm is public. > This report essentially says "if we ran Google, we could get better CTR's by showing Yelp results". That does not prove consumer harm. The report says that they believed that Google was suppressing results from "third-party review sites such as Yelp and TripAdvisor" (not just Yelp) that it would have otherwise deemed to be relevant, in order to prioritize Google's own services, and that this artificial adjustment of the search results is harming consumers because it's yielding results that are less useful to consumers (as measured by CTR). Which is a bit different than what you're saying. And this actually makes sense to me. When I search on DuckDuckGo for places in my city, I often get yelp results near or at the top. So they're obviously deemed relevant by the unbiased ranking that DDG uses. And I often get Yelp results if I search on Google as well. So obviously Google thinks Yelp is relevant for many searches. So if I perform a search that Yelp has relevant results for, and Google isn't showing me those results but instead shows me e.g. a Google+ business listing, then it seems quite reasonable to claim that Google is artificially suppressing the Yelp results in that case, and to further claim that this is causing consumer harm because the Yelp results were more relevant than what Google chose to show instead.

by eridius 11y ago

downandout 11y ago

One screenshot showed Google's results, while another shows search results that were purportedly sorted purely by relevance. Therein lies the problem. They don't know how Google would order the results in the hypothetical scenario where results didn't include those that make Google money. This goes beyond not having clickthrough data; what they were showing to their test users was simply a not-very-good, human interpretation of what they presume the Google algorithm might return in a nonexistent situation. Unsurprisingly, these fictional search results had worse CTR's than were achieved by displaying presumably custom-crafted Yelp results. The term relevance is unique to each search algorithm, and these researchers did not have the data to determine what the relevance would otherwise be for Google. This report essentially says "if we ran Google, we could get better CTR's by showing Yelp results". That does not prove consumer harm.

eridius 11y ago
> They don't know how Google would order the results in the hypothetical scenario where results didn't include those that make Google money. They kind of do. Google's very public about most of how they sort results, and the article said they used Google's "algorithm" to rank them. I know Google used to even expose the underlying PageRank number for various results, but I don't know if they still do so today (if so, then it becomes even easier to sort them the way you think Google would). > what they were showing to their test users was simply a not-very-good, human interpretation of what they presume the Google algorithm might return in a nonexistent situation You're making a very bad assumption here. From the article: > One set of users saw a page reflecting results currently displayed by Google Based on that it sure sounds like they actually performed various searches on Google, screenshotted the results, then fabricated a second set of results that reflects what they believed Google would have shown if it sorted purely by "relevance" instead of prioritizing its own services. And these fabricated results had significantly higher CTRs than the genuine results from Google. > The term relevance is unique to each search algorithm, and these researchers did not have the data to determine what the relevance would otherwise be for Google. Sure they do. The way Google ranks search results is quite public and has been so from the beginning. What isn't public is all the various tweaks they perform to the results, such as removing what they believe to be spam or content farms, as well as the biasing they do in favor of their own services. But the core algorithm is public. > This report essentially says "if we ran Google, we could get better CTR's by showing Yelp results". That does not prove consumer harm. The report says that they believed that Google was suppressing results from "third-party review sites such as Yelp and TripAdvisor" (not just Yelp) that it would have otherwise deemed to be relevant, in order to prioritize Google's own services, and that this artificial adjustment of the search results is harming consumers because it's yielding results that are less useful to consumers (as measured by CTR). Which is a bit different than what you're saying. And this actually makes sense to me. When I search on DuckDuckGo for places in my city, I often get yelp results near or at the top. So they're obviously deemed relevant by the unbiased ranking that DDG uses. And I often get Yelp results if I search on Google as well. So obviously Google thinks Yelp is relevant for many searches. So if I perform a search that Yelp has relevant results for, and Google isn't showing me those results but instead shows me e.g. a Google+ business listing, then it seems quite reasonable to claim that Google is artificially suppressing the Yelp results in that case, and to further claim that this is causing consumer harm because the Yelp results were more relevant than what Google chose to show instead.