4 ms·

It's great that they did this study. I wish more people would publicly challenge many of the outlandish claims in advertising. But it seems rather hypocritical

by pyrocat 13y ago

It's great that they did this study. I wish more people would publicly challenge many of the outlandish claims in advertising. But it seems rather hypocritical to knock Microsoft for a sample size of "nearly 1,000" when their own study "obtained 1,008 Bing It On challenge responses from the MTurk platform and narrowed our analysis to 985 respondents who submitted screen shots for 4925 searches."

yid 13y ago
Not in this case, no. They were trying to replicate Microsoft's results, which entails using a similar sample size.
- justinsb 13y ago
  I don't think that's quite right. Imagine if I flip a coin and it comes up heads, so I say "coins always come up heads". You don't flip a coin once to prove me wrong. I think the right thing to do is to conduct a much bigger study, get a much better estimate of the preference distribution, and then derive the % probability of getting a result at least as favorable as the Bing result by pure chance. If you can show there's only e.g. a 1% chance that the Bing result arose through random variation, then Bing definitely has some tough questions to answer. (e.g. did they run 100 different studies, and just cherry-picked the best result).
  - yid 13y ago
    Yes, that would have been a great way to do it as well. However, the most interesting aspect of the study to me was that they used Microsoft's methodology, comparable sample sizes, and examined the effect of changing search terms from Microsoft's "recommended" list to ones that study participants suggested themselves. Incidentally, you don't need a giant sample to estimate the variance on the Google/Bing ratio -- you can use any number of resampling techniques like bootstrapping to get that estimate.
    - justinsb 13y ago
      Good point - particularly as it's a fairly simple universe of answers. I wonder if we could get the raw data. But, I do think the Mechanical Turk vs "shopping mall" selection-bias factor overwhelms everything else.
- mattwallaert 13y ago
  Except they split the sample into three, thereby reducing the sample size (and power) in thirds. What matters is that there are still large enough to run significance tests on, both in the original claim studies and Ayers' study. (Note: I work for Bing.)

by pyrocat 13y ago

yid 13y ago

Not in this case, no. They were trying to replicate Microsoft's results, which entails using a similar sample size.

justinsb 13y ago
I don't think that's quite right. Imagine if I flip a coin and it comes up heads, so I say "coins always come up heads". You don't flip a coin once to prove me wrong. I think the right thing to do is to conduct a much bigger study, get a much better estimate of the preference distribution, and then derive the % probability of getting a result at least as favorable as the Bing result by pure chance. If you can show there's only e.g. a 1% chance that the Bing result arose through random variation, then Bing definitely has some tough questions to answer. (e.g. did they run 100 different studies, and just cherry-picked the best result).
- yid 13y ago
  Yes, that would have been a great way to do it as well. However, the most interesting aspect of the study to me was that they used Microsoft's methodology, comparable sample sizes, and examined the effect of changing search terms from Microsoft's "recommended" list to ones that study participants suggested themselves. Incidentally, you don't need a giant sample to estimate the variance on the Google/Bing ratio -- you can use any number of resampling techniques like bootstrapping to get that estimate.
  - justinsb 13y ago
    Good point - particularly as it's a fairly simple universe of answers. I wonder if we could get the raw data. But, I do think the Mechanical Turk vs "shopping mall" selection-bias factor overwhelms everything else.
mattwallaert 13y ago
Except they split the sample into three, thereby reducing the sample size (and power) in thirds. What matters is that there are still large enough to run significance tests on, both in the original claim studies and Ayers' study. (Note: I work for Bing.)