The problem with this study is that the tested search queries are completely different than what people type into a search engine. Besides being 30% vile, real

The problem with this study is that the tested search queries are completely different than what people type into a search engine. Besides being 30% vile, real search queries are very tailored to search engines algorithms -- two to three words, maximal idf (inverse document frequency). People type in the minimum number of least commonly used terms that distinguish what they want. Both Google and Bing use human evaluators (sometimes with domain expertise) to evaluate real queries against the search results. None of those real queries look anything like these. Unfortunately, their hasn't been a good public release of search engine query data since the AOL fiasco.