7 ms·

Halfway down that piece: A study this year by Stone Temple, a prominent analyst of the industry, showed Google’s search engine answered 74.3% of 5,000 question

Halfway down that piece: A study this year by Stone Temple, a prominent analyst of the industry, showed Google’s search engine answered 74.3% of 5,000 questions, and on those answers it had a 97.4% accuracy rate. Both percentages are higher than services from Amazon.com Inc., Apple Inc. and Microsoft Corp. Alternative WSJ: Let's bury that tidbit under the fold while we nitpick and highlight edge cases in a constantly improving system. We should also emphasising how integral google's search is to the health of society and civilization because we are mandated by management and ownership to produce anti-google PR and provide a steady supply of ammunition to its critics.

maxerickson 9y ago
97% accuracy is abysmal for something you are presenting as "answers".
- Gigablah 9y ago
  It’s good enough for my professor :)
- johnhenry 9y ago
  Not sure what the margin of error is on these things, but generally 97% should probably be considered 100%.
  - maxerickson 9y ago
    There's no margin of error for the test they did, just potential sampling bias. Given the 5000 questions they asked, the system will provide the wrong answer 2.6% of the time. Every time, until they improve it. There's a chance that they managed to ask the only 150 questions that it doesn't know the answer to, but not a very big one.
    - johnhenry 9y ago
      I agree -- given the sample space, there is not effective way to calculate the margin of error. Even so, I don't think that there are many examples of non-deterministic mechanisms that produce the correct answer 97% of the time.
  - deleted 9y ago
    [deleted]
  - jgalt212 9y ago
    a parser that is 97% correct is broken.
    - Kurtz79 9y ago
      A parser (assuming you are talking about a programming language parser) has the luxury of having highly structured and deterministic inputs, and to be able to refuse giving an "answer" if they are not.
      - jgalt212 9y ago
        > highly structured and deterministic inputs not all parsers deal with such inputs, and parsers that don't can and often do produce multiple interpretations of the input data.
        Kurtz79 9y ago
        Tell me of some parsers that do not deal with deterministic inputs and have 100% accuracy, then.
    - astrodust 9y ago
      It's like Intel's floating point math used to be about 97% correct. Good enough, right? https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
- retSava 9y ago
  What you/we need is a healthy dose of taking things with a grain of salt and not just blindly trust what we read/hear. This goes not only for results in search services today, but was also valid in the time before this, when we used encyclopedias, or asked someone.
  - maxerickson 9y ago
    I'm not sure why you feel the need to comment on how credulous I might be. Anyway, the 97% is an impressive result, it just isn't so impressive as to be beyond criticism. My comment is a little over the top, in response to the other comment that sets the bar at effusive praise being the only proper analysis of the system.
  - astrodust 9y ago
    This is not about "a grain of salt". Imagine you're a ten year old kid and you're looking for answers on some subject, say the Holocaust. What if all the "answers" you're getting are from far-right hate groups? How do you think that will affect that person? Likewise, if you're looking for medical advice and you're getting nothing but anti-science woo, can we really be surprised that anti-vaxxers are becoming more numerous? These outbreaks of deadly, yet preventable diseases have serious fatal consequences for many. "Grain of salt" means fuck all when people are dying from bad answers. You can't learn from your mistakes when you're dead. At least Google could frame these with the idea of keeping a skeptical mind, but they should probably stop surfacing things as "the answer" and instead as "top search result".
- yathern 9y ago
  If there was someone I knew who I could ask a question, with a 97% chance they were correct, I'd consider them to be a phenomenon, and an extremely valuable resource. For some definition of correct, I'd be happy to be 60% accurate.
  - maxerickson 9y ago
    Yeah but what if you got to choose between 2 people that knew the answer 50% of the time but one of the them enjoyed making up answers when they didn't know and the other just told you they didn't know?
NaliSauce 9y ago
I dug up the study by Stone Temple to see what kind of questions were asked (eg: are the questions really trivial or not) and they provide a few exampled in their write up. I am actually seriously impressed by how well those "personal assistants" work compared to what I would expect. Kudos to the engineers who built those things! (Disclosure: I never worked on any of those, nor do I work for any of the companies making them)
untog 9y ago
That's absolutely the wrong way to go about looking at it. Not all facts are of equal importance. 97 correct facts about, say, gardening, and three incorrect facts about first aid is not a successful system. If Google is presenting the tool as a canonical source of truth, it needs to be right 100% of the time. In fact, being right 97.4% of the time is worse in many ways, as it lulls people into a false sense of security about how much they can trust the system. Get 97/100 obvious answers correct, then give someone wildly inappropriate advice when they ask something more "off piste". And let's not delude ourselves here, Google isn't doing this out of an altrustic desire to help people. They're making sure fewer people leave google.com, in the process starving the very sites they're getting information from of revenue. That should worry all of us. > we are mandated by management and ownership to produce anti-google PR and provide a steady supply of ammunition to its critics. This feels like an absurd criticism. Couldn't it apply to absolutely every negative thing written about Google, ever? Why should Google be immune from criticism?
- pesenti 9y ago
  And that’s a ridiculous standard because there is no such thing as 100% when talking about knowledge. Humans and experts wouldn’t agree 100% of the time so it’s impossible for computers to get there.
  - untog 9y ago
    So why create a feature that pretends such a thing is possible, and promote it as heavily as Google does? In many ways, "I'm feeling lucky" was a great middle ground - by clicking it you're implictly saying "I know this might not actually be what I'm looking for, but I'm willing to compromise". Google has taken that compromise and stuck it at the top of the search results page.
    - RyanZAG 9y ago
      For profit corporations require that the senior employees they hire for impossibly high paychecks are constantly providing benefits that can be brought up in quarterly board meetings. This means taking an already functional product and making changes to it -- any change -- that can be spun as being a positive improvement worthy of a promotion. Google Search is already where it should be. Yet Google Search employs thousands of incredibly expensive engineers. Something has to be done, and that "something" is rarely going to be good, and almost guaranteed to be something nobody actually wanted.
dep_b 9y ago
Google used to be a place (when it was really new) where you would always find what you were looking for within the 10 first responses because of the power of it's "Pigeon Ranking". I'm not claiming 97.4% in all cases but really often "I Feel Lucky" would be the right answer. Slowly this got gamed by people that wanted to have their sites there and this magic disappeared. I imagine these selected answers will either come for a dwindling amount of sources of "truth", turning Google in a sort of Yahoo! with lots of handpicked results or again these algorithms will be gamed and we get the same mess as Google turned into where it was constantly fighting new waves of spam.
devy 9y ago
Philosophically speaking, assuming there is always a single answer for any question and promoting a specific SINGLE answer for them is not only inaccurate and misleading, sometime it's WRONG! The world is complex, many questions will and should have multiple answers depending how you look at it.

maxerickson 9y ago

97% accuracy is abysmal for something you are presenting as "answers".

Gigablah 9y ago
It’s good enough for my professor :)
johnhenry 9y ago
Not sure what the margin of error is on these things, but generally 97% should probably be considered 100%.
- maxerickson 9y ago
  There's no margin of error for the test they did, just potential sampling bias. Given the 5000 questions they asked, the system will provide the wrong answer 2.6% of the time. Every time, until they improve it. There's a chance that they managed to ask the only 150 questions that it doesn't know the answer to, but not a very big one.
  - johnhenry 9y ago
    I agree -- given the sample space, there is not effective way to calculate the margin of error. Even so, I don't think that there are many examples of non-deterministic mechanisms that produce the correct answer 97% of the time.
- deleted 9y ago
  [deleted]
- jgalt212 9y ago
  a parser that is 97% correct is broken.
  - Kurtz79 9y ago
    A parser (assuming you are talking about a programming language parser) has the luxury of having highly structured and deterministic inputs, and to be able to refuse giving an "answer" if they are not.
    - jgalt212 9y ago
      > highly structured and deterministic inputs not all parsers deal with such inputs, and parsers that don't can and often do produce multiple interpretations of the input data.
      - Kurtz79 9y ago
        Tell me of some parsers that do not deal with deterministic inputs and have 100% accuracy, then.
  - astrodust 9y ago
    It's like Intel's floating point math used to be about 97% correct. Good enough, right? https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
retSava 9y ago
What you/we need is a healthy dose of taking things with a grain of salt and not just blindly trust what we read/hear. This goes not only for results in search services today, but was also valid in the time before this, when we used encyclopedias, or asked someone.
- maxerickson 9y ago
  I'm not sure why you feel the need to comment on how credulous I might be. Anyway, the 97% is an impressive result, it just isn't so impressive as to be beyond criticism. My comment is a little over the top, in response to the other comment that sets the bar at effusive praise being the only proper analysis of the system.
- astrodust 9y ago
  This is not about "a grain of salt". Imagine you're a ten year old kid and you're looking for answers on some subject, say the Holocaust. What if all the "answers" you're getting are from far-right hate groups? How do you think that will affect that person? Likewise, if you're looking for medical advice and you're getting nothing but anti-science woo, can we really be surprised that anti-vaxxers are becoming more numerous? These outbreaks of deadly, yet preventable diseases have serious fatal consequences for many. "Grain of salt" means fuck all when people are dying from bad answers. You can't learn from your mistakes when you're dead. At least Google could frame these with the idea of keeping a skeptical mind, but they should probably stop surfacing things as "the answer" and instead as "top search result".
yathern 9y ago
If there was someone I knew who I could ask a question, with a 97% chance they were correct, I'd consider them to be a phenomenon, and an extremely valuable resource. For some definition of correct, I'd be happy to be 60% accurate.
- maxerickson 9y ago
  Yeah but what if you got to choose between 2 people that knew the answer 50% of the time but one of the them enjoyed making up answers when they didn't know and the other just told you they didn't know?

NaliSauce 9y ago

I dug up the study by Stone Temple to see what kind of questions were asked (eg: are the questions really trivial or not) and they provide a few exampled in their write up. I am actually seriously impressed by how well those "personal assistants" work compared to what I would expect. Kudos to the engineers who built those things! (Disclosure: I never worked on any of those, nor do I work for any of the companies making them)

untog 9y ago

That's absolutely the wrong way to go about looking at it. Not all facts are of equal importance. 97 correct facts about, say, gardening, and three incorrect facts about first aid is not a successful system. If Google is presenting the tool as a canonical source of truth, it needs to be right 100% of the time. In fact, being right 97.4% of the time is worse in many ways, as it lulls people into a false sense of security about how much they can trust the system. Get 97/100 obvious answers correct, then give someone wildly inappropriate advice when they ask something more "off piste". And let's not delude ourselves here, Google isn't doing this out of an altrustic desire to help people. They're making sure fewer people leave google.com, in the process starving the very sites they're getting information from of revenue. That should worry all of us. > we are mandated by management and ownership to produce anti-google PR and provide a steady supply of ammunition to its critics. This feels like an absurd criticism. Couldn't it apply to absolutely every negative thing written about Google, ever? Why should Google be immune from criticism?

pesenti 9y ago
And that’s a ridiculous standard because there is no such thing as 100% when talking about knowledge. Humans and experts wouldn’t agree 100% of the time so it’s impossible for computers to get there.
- untog 9y ago
  So why create a feature that pretends such a thing is possible, and promote it as heavily as Google does? In many ways, "I'm feeling lucky" was a great middle ground - by clicking it you're implictly saying "I know this might not actually be what I'm looking for, but I'm willing to compromise". Google has taken that compromise and stuck it at the top of the search results page.
  - RyanZAG 9y ago
    For profit corporations require that the senior employees they hire for impossibly high paychecks are constantly providing benefits that can be brought up in quarterly board meetings. This means taking an already functional product and making changes to it -- any change -- that can be spun as being a positive improvement worthy of a promotion. Google Search is already where it should be. Yet Google Search employs thousands of incredibly expensive engineers. Something has to be done, and that "something" is rarely going to be good, and almost guaranteed to be something nobody actually wanted.

dep_b 9y ago

Google used to be a place (when it was really new) where you would always find what you were looking for within the 10 first responses because of the power of it's "Pigeon Ranking". I'm not claiming 97.4% in all cases but really often "I Feel Lucky" would be the right answer. Slowly this got gamed by people that wanted to have their sites there and this magic disappeared. I imagine these selected answers will either come for a dwindling amount of sources of "truth", turning Google in a sort of Yahoo! with lots of handpicked results or again these algorithms will be gamed and we get the same mess as Google turned into where it was constantly fighting new waves of spam.

devy 9y ago

Philosophically speaking, assuming there is always a single answer for any question and promoting a specific SINGLE answer for them is not only inaccurate and misleading, sometime it's WRONG! The world is complex, many questions will and should have multiple answers depending how you look at it.