3 ms·
> For the current research, a set of 30 “core” news questions was developed Right. Let's talk about statistics for a bit. Or let's put it differently: they fou
by menaerus 1y ago
> For the current research, a set of 30 “core” news questions was developed
Right. Let's talk about statistics for a bit. Or let's put it differently: they found in their report that 45% of the answers for 30 questions they have "developed" had a significant issue, e.g. inexisting reference
I'll give you 30 questions out of my sleeve where 95% of the answers will not have any significant issue.
- matthewmacleod 1y agoYes, I'm sure you could hack together some bullshit questions to demonstrate whatever you want. Is there a specific reason that the reasonably straightforward methodology they did use is somehow flawed?
- menaerus 1y agoYes, and you answered it yourself.
- darkwater 1y agoErr, no? Being _possible_ does not necessarily imply that's what happened.
- menaerus 1y agoA bucket of 30 questions is not a statistically significant sample size which we can use to support the hypothesis which goes to say that all AI assistants they tested are 45% of the time wrong. That's not how science works. Neither is my bucket of 30 questions statistcally significant but it goes to say that I can disprove their hypothesis just by giving them my sample. I think that the report is being disingenious and I don't understand for what reasons. it's funny that they say "misrepresent" when that's exactly what they are doing.
- extrabajs 1y agoStatistically significant... sample size? Support the hypothesis?
- frm88 1y agoI don't follow your reasoning re. statistical sample size. The topic article claims that 45% of the answers were wrong. If - with a vastly greater sample size - the answers were "only" (let's say) 20% wrong, that's still a complete failure, so is 5%. The article is not about hypothesis, it's about news reporting.