5 ms·

Matt's response to this: http://www.google.com/buzz/109412257237874861202/A7d5joZ3tJJ/This-isnt-worth-a-full-blog-post-but-Ill-jot-a http://www.google.com/buzz/

Matt's response to this: http://www.google.com/buzz/109412257237874861202/A7d5joZ3tJJ/This-isnt-worth-a-full-blog-post-but-Ill-jot-a http://www.google.com/buzz/109412257237874861202/A7d5joZ3tJJ...

nostrademons 16y ago
There's another possible source of selection bias: users may be going to Google for queries that they know are a hard, because they assume that Bing won't have the answer. Of course such queries will fail more often, because they're hard. Bing's marketing has centered around the head of the query distribution - common tasks where definite answers exist, like how to book a flight. It makes sense that such tasks would succeed much more often than long-tail ones like looking up an obscure error message.
- barista 16y ago
  Well if they are "going" to Google for hard queries then wouldn't they try the queries on Bing first? Or are you saying that people spend time deciding a query is simple or hard and then choose appropriate search engine for that?
  - amalcon 16y ago
    They might try the query on Bing, fail, and then try the same query on Google. A large portion of those are likely to fail as well, just because the queries are hard. It's less likely that someone will try a query on Google, fail, and then think "Maybe Bing will do a better job with this."
    - barista 16y ago
      Well in that case wouldn't bing's percentage of failed searches be even larger than Google's? The stats don't reflect that
      - amalcon 16y ago
        It would cause Google to have a relatively larger proportion of "difficult" searches. Many "difficult" searches on Bing will be repeated on Google, but not the reverse. If you assume that all "difficult" searches fail on both engines as a first approximation, that would mean that Google will fail all the "difficult" searches, while Bing will only fail the "difficult" searches made by people who primarily use Bing.
  - nostrademons 16y ago
    More that they have certain problem domains where they might pick a given search engine, and that those problem domains are biased toward using Bing for easy searches and Google for hard ones. I'll frequently search for airplane tickets on Bing, for example, while I won't even go to Google for that. But when it comes to looking up obscure programming stuff, I'm not going to bother searching on Bing, I know it'll suck.
    - jamesbritt 16y ago
      "More that they have certain problem domains where they might pick a given search engine, and that those problem domains are biased toward using Bing for easy searches and Google for hard ones." That makes no sense. Why would you, or anyone, alternate with search engines if one of them too often fails to deliver? Why not just use Google all the time and avoid the cognitive overhead? Does Google do a poor job, or a worse than Bing job, with airplane tickets?
      - rue 16y ago
        I use DDG except for local sites, which it doesn't index anywhere near as well as Unca Googs.
- notahacker 16y ago
  A third possible source would be bias in user samples: If Bing users are on average less sophisticated (plausible given the assumption that less sophisticated users tend to use IE with the default search engine) they are more likely to click through irrespective of search results' relevance or the potential to improve a query. What does the "success rate" of AOL's Google-powered search look like?
jdp23 16y ago
From a PR standpoint, the headlines here make it look like a big win for MS http://is.gd/b92NRz http://is.gd/b92NRz
natmaster 16y ago
Does someone have a link to the information on how Experian's data was gathered? It looks to me like 'success' is defined as actually clicking a result - not the ridiculous things like switching to google - but their explanation on the report page is fairly nebulous.
- Matt_Cutts 16y ago
  Suppose the user was on search engine A and then goes to site B. I believe Hitwise has no way of knowing whether the user clicked on a link in the search results to go to site B vs. just going to site B on their own. As far as how the data was gathered, Hitwise gets clickstream data from ISPs: http://www.hitwise.com/us/about-us/how-we-do-it http://www.hitwise.com/us/about-us/how-we-do-it
barista 16y ago
Here's the gist of Matt's reply: 1. Hitwise's definition of a successful search doesn't actually involve clicking on result. 2. MSN has pages that generate bing search while masquerading as regular pages.
brudgers 16y ago
Mr. Cutts provides an example of Kim Kardashian videos on MSN linking to Bing as an explanation for the difference in success ratings between Bing and Google. However if that were a significant factor, the similarity between the ratings for Bing and Yahoo would be somewhat unexpected without a demonstration that similar searches are directed to Yahoo. It would also seem that there is probably a significant enough behavioral difference between the user bases of Bing and Yahoo Search for Matt's "abandoned search theory" to be consistent with the similarity in success rates between the sites. A more plausible explanation (but "creepier" in Googlespeak) for Google's results is that Google's sometimes practice of not linking directly to sites but rather to an internal redirecting page [as described by jacques mattheij here: http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+some+observations http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+...] lowers Google's success rate. It would also explain why Google questions the value of the metric so vigorously and why their criticism is speculative and anecdotal rather than more scientifically argued. http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+some+observations http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+...
bkhl 16y ago
I clearly see where he is coming from and I also believe that the metric Hitwise used is just BS...BUT! the more Matt becomes sensitive about Bing, the more pageviews Bing would get. I bet Bing got quite unexpected PR recently due to Matt's accusation (I don't want to talk about the issue since my opinion is mine). I'm curious to see metrics for Feb. ;)
happybuy 16y ago
Very interesting that the first response most HN commenters have when confronted with data that Bing provides better search results than Google is to doubt the data. This despite that fact that (I'm guessing) most commenters have not actually extensively used Bing day in and day out. Shows the huge barrier to entry that any new search engine has: even if they provide measurably better search results even tech savvy users have an almost unshakeable belief that Google results will always or somehow must be better. There seems to be some infallible belief that Google provides better search results, always will and that there is some 'magic' Google has for search that no other company could possibly reproduce or exceed. There is no logical reason that this should be the case and in many ways its more logical to suggest that Google is so wed to its current techniques and revenue models that it has no incentive to disrupt itself. The recent furore over spam (and Adsense which funds its existence) perhaps is the first evidence that this is actually the case. Google made some noise about rectifying the spam issue but really they didn't make any hard decisions in this space. A personal anecdote - I switched all my browsers and iPhone to default to Bing about a month ago - the experience has been pleasant, the results excellent and haven't found the need or desire to use Google search since.
- chalst 16y ago
  Very interesting that the first response most HN commenters have when confronted with data that Bing provides better search results than Google is to doubt the data. The very first reaction I have to most studies is to question first the methodology, then the data. Remember that 88.2% of statistics are made up on the spot.
  - happybuy 16y ago
    Yes it's true that people are rightly doubting the methodology. However if the data showed the exact opposite result (Google having better results than Bing) I think the reaction on HN would be to accept the results at face value. This despite the fact that there is no reason (that I'm aware of) that Google search results should be inherently better than Bing. In this instance, I think it shows the strength of the Google brand and how the strength of that brand sets users expectations for the quality of the results.

Matt's response to this: http://www.google.com/buzz/109412257237874861202/A7d5joZ3tJJ/This-isnt-worth-a-full-blog-post-but-Ill-jot-a http://www.google.com/buzz/109412257237874861202/A7d5joZ3tJJ...

nostrademons 16y ago

There's another possible source of selection bias: users may be going to Google for queries that they know are a hard, because they assume that Bing won't have the answer. Of course such queries will fail more often, because they're hard. Bing's marketing has centered around the head of the query distribution - common tasks where definite answers exist, like how to book a flight. It makes sense that such tasks would succeed much more often than long-tail ones like looking up an obscure error message.

barista 16y ago
Well if they are "going" to Google for hard queries then wouldn't they try the queries on Bing first? Or are you saying that people spend time deciding a query is simple or hard and then choose appropriate search engine for that?
- amalcon 16y ago
  They might try the query on Bing, fail, and then try the same query on Google. A large portion of those are likely to fail as well, just because the queries are hard. It's less likely that someone will try a query on Google, fail, and then think "Maybe Bing will do a better job with this."
  - barista 16y ago
    Well in that case wouldn't bing's percentage of failed searches be even larger than Google's? The stats don't reflect that
    - amalcon 16y ago
      It would cause Google to have a relatively larger proportion of "difficult" searches. Many "difficult" searches on Bing will be repeated on Google, but not the reverse. If you assume that all "difficult" searches fail on both engines as a first approximation, that would mean that Google will fail all the "difficult" searches, while Bing will only fail the "difficult" searches made by people who primarily use Bing.
- nostrademons 16y ago
  More that they have certain problem domains where they might pick a given search engine, and that those problem domains are biased toward using Bing for easy searches and Google for hard ones. I'll frequently search for airplane tickets on Bing, for example, while I won't even go to Google for that. But when it comes to looking up obscure programming stuff, I'm not going to bother searching on Bing, I know it'll suck.
  - jamesbritt 16y ago
    "More that they have certain problem domains where they might pick a given search engine, and that those problem domains are biased toward using Bing for easy searches and Google for hard ones." That makes no sense. Why would you, or anyone, alternate with search engines if one of them too often fails to deliver? Why not just use Google all the time and avoid the cognitive overhead? Does Google do a poor job, or a worse than Bing job, with airplane tickets?
    - rue 16y ago
      I use DDG except for local sites, which it doesn't index anywhere near as well as Unca Googs.
notahacker 16y ago
A third possible source would be bias in user samples: If Bing users are on average less sophisticated (plausible given the assumption that less sophisticated users tend to use IE with the default search engine) they are more likely to click through irrespective of search results' relevance or the potential to improve a query. What does the "success rate" of AOL's Google-powered search look like?

jdp23 16y ago

From a PR standpoint, the headlines here make it look like a big win for MS http://is.gd/b92NRz http://is.gd/b92NRz

natmaster 16y ago

Does someone have a link to the information on how Experian's data was gathered? It looks to me like 'success' is defined as actually clicking a result - not the ridiculous things like switching to google - but their explanation on the report page is fairly nebulous.

Matt_Cutts 16y ago
Suppose the user was on search engine A and then goes to site B. I believe Hitwise has no way of knowing whether the user clicked on a link in the search results to go to site B vs. just going to site B on their own. As far as how the data was gathered, Hitwise gets clickstream data from ISPs: http://www.hitwise.com/us/about-us/how-we-do-it http://www.hitwise.com/us/about-us/how-we-do-it

barista 16y ago

Here's the gist of Matt's reply: 1. Hitwise's definition of a successful search doesn't actually involve clicking on result. 2. MSN has pages that generate bing search while masquerading as regular pages.

brudgers 16y ago

Mr. Cutts provides an example of Kim Kardashian videos on MSN linking to Bing as an explanation for the difference in success ratings between Bing and Google. However if that were a significant factor, the similarity between the ratings for Bing and Yahoo would be somewhat unexpected without a demonstration that similar searches are directed to Yahoo. It would also seem that there is probably a significant enough behavioral difference between the user bases of Bing and Yahoo Search for Matt's "abandoned search theory" to be consistent with the similarity in success rates between the sites. A more plausible explanation (but "creepier" in Googlespeak) for Google's results is that Google's sometimes practice of not linking directly to sites but rather to an internal redirecting page [as described by jacques mattheij here: http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+some+observations http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+...] lowers Google's success rate. It would also explain why Google questions the value of the metric so vigorously and why their criticism is speculative and anecdotal rather than more scientifically argued. http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+some+observations http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+...

bkhl 16y ago

I clearly see where he is coming from and I also believe that the metric Hitwise used is just BS...BUT! the more Matt becomes sensitive about Bing, the more pageviews Bing would get. I bet Bing got quite unexpected PR recently due to Matt's accusation (I don't want to talk about the issue since my opinion is mine). I'm curious to see metrics for Feb. ;)

happybuy 16y ago

Very interesting that the first response most HN commenters have when confronted with data that Bing provides better search results than Google is to doubt the data. This despite that fact that (I'm guessing) most commenters have not actually extensively used Bing day in and day out. Shows the huge barrier to entry that any new search engine has: even if they provide measurably better search results even tech savvy users have an almost unshakeable belief that Google results will always or somehow must be better. There seems to be some infallible belief that Google provides better search results, always will and that there is some 'magic' Google has for search that no other company could possibly reproduce or exceed. There is no logical reason that this should be the case and in many ways its more logical to suggest that Google is so wed to its current techniques and revenue models that it has no incentive to disrupt itself. The recent furore over spam (and Adsense which funds its existence) perhaps is the first evidence that this is actually the case. Google made some noise about rectifying the spam issue but really they didn't make any hard decisions in this space. A personal anecdote - I switched all my browsers and iPhone to default to Bing about a month ago - the experience has been pleasant, the results excellent and haven't found the need or desire to use Google search since.

chalst 16y ago
Very interesting that the first response most HN commenters have when confronted with data that Bing provides better search results than Google is to doubt the data. The very first reaction I have to most studies is to question first the methodology, then the data. Remember that 88.2% of statistics are made up on the spot.
- happybuy 16y ago
  Yes it's true that people are rightly doubting the methodology. However if the data showed the exact opposite result (Google having better results than Bing) I think the reaction on HN would be to accept the results at face value. This despite the fact that there is no reason (that I'm aware of) that Google search results should be inherently better than Bing. In this instance, I think it shows the strength of the Google brand and how the strength of that brand sets users expectations for the quality of the results.