4 ms·
97% accuracy is abysmal for something you are presenting as "answers".
by maxerickson 9y ago
97% accuracy is abysmal for something you are presenting as "answers".
- Gigablah 9y agoIt’s good enough for my professor :)
- johnhenry 9y agoNot sure what the margin of error is on these things, but generally 97% should probably be considered 100%.
- maxerickson 9y agoThere's no margin of error for the test they did, just potential sampling bias. Given the 5000 questions they asked, the system will provide the wrong answer 2.6% of the time. Every time, until they improve it. There's a chance that they managed to ask the only 150 questions that it doesn't know the answer to, but not a very big one.
- johnhenry 9y agoI agree -- given the sample space, there is not effective way to calculate the margin of error. Even so, I don't think that there are many examples of non-deterministic mechanisms that produce the correct answer 97% of the time.
- deleted 9y ago[deleted]
- jgalt212 9y agoa parser that is 97% correct is broken.
- Kurtz79 9y agoA parser (assuming you are talking about a programming language parser) has the luxury of having highly structured and deterministic inputs, and to be able to refuse giving an "answer" if they are not.
- jgalt212 9y ago> highly structured and deterministic inputs not all parsers deal with such inputs, and parsers that don't can and often do produce multiple interpretations of the input data.
- Kurtz79 9y agoTell me of some parsers that do not deal with deterministic inputs and have 100% accuracy, then.
- astrodust 9y agoIt's like Intel's floating point math used to be about 97% correct. Good enough, right? https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
- retSava 9y agoWhat you/we need is a healthy dose of taking things with a grain of salt and not just blindly trust what we read/hear. This goes not only for results in search services today, but was also valid in the time before this, when we used encyclopedias, or asked someone.
- maxerickson 9y agoI'm not sure why you feel the need to comment on how credulous I might be. Anyway, the 97% is an impressive result, it just isn't so impressive as to be beyond criticism. My comment is a little over the top, in response to the other comment that sets the bar at effusive praise being the only proper analysis of the system.
- astrodust 9y agoThis is not about "a grain of salt". Imagine you're a ten year old kid and you're looking for answers on some subject, say the Holocaust. What if all the "answers" you're getting are from far-right hate groups? How do you think that will affect that person? Likewise, if you're looking for medical advice and you're getting nothing but anti-science woo, can we really be surprised that anti-vaxxers are becoming more numerous? These outbreaks of deadly, yet preventable diseases have serious fatal consequences for many. "Grain of salt" means fuck all when people are dying from bad answers. You can't learn from your mistakes when you're dead. At least Google could frame these with the idea of keeping a skeptical mind, but they should probably stop surfacing things as "the answer" and instead as "top search result".
- yathern 9y agoIf there was someone I knew who I could ask a question, with a 97% chance they were correct, I'd consider them to be a phenomenon, and an extremely valuable resource. For some definition of correct, I'd be happy to be 60% accurate.
- maxerickson 9y agoYeah but what if you got to choose between 2 people that knew the answer 50% of the time but one of the them enjoyed making up answers when they didn't know and the other just told you they didn't know?