2 ms·

I thought this was an interesting study but, after spending a few minutes trying to find patterns in the data, I finally hacked up a quick null hypothesis graph

• lukas 17y ago
  Hey - I left a post on your blog, but I don't agree with your assessment. Generating a similar shape with a random process doesn't imply that our data is has no signal, and if you actually lay out the graphs side by side you'll notice that our graph is somewhat shifted to the right.
• etal 17y ago
  The shape of this kind of graph usually isn't very meaningful by itself for showing subtle patterns -- you're right that the extremes of the graph always look like that. In the original article, if you line up the Bing-vs-Google and Bing-vs-Live graphs, the intercept near the middle of the graph is a little further left. That's all I got from it. I'm assuming there's an ANOVA table associated with this study that we're not seeing, and the probabilities there must be a little more compelling. Maybe a two-phase Turking might be interesting, too: first have the turkers fill come up with a few parameters for each query (e.g. current events, general info, celebrities, pr0n), then compare search engine results for those queries. That would help pick out the more subtle patterns that you were looking for in the original graph.
  • shalmanese 17y ago
    With only 6 to 8 participants, the study definitely didn't have the power to find all but the grossest differences between query types. It's not about study design so much as the need to recruit more people.