8 ms·
Most leading chatbots routinely exaggerate science findings
- gustavus 1y ago"Over a year, we collected 4,900 summaries. When we analysed them, we found that six of ten models systematically exaggerated claims they found in the original texts" So it turns out llms trained largely on Internet science articles make the same mistakes as are made by science journalists.
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- bethekidyouwant 1y agoMost leading chatbots routinely exaggerate science findings - ahh yes this is completely unique to llms
- kjfaejgoaeijhei 1y agoTo quote the article: "Chatbots were nearly five times more likely to produce broad generalisations than their human counterparts." The following might be rude and unkind, but at this point frankly necessary: LLM apologists should stop projecting.
- redcobra762 1y agoWhat is the human counterpart to an LLM? The article doesn't describe. And "LLM apologists" is so polemical it's hard to take seriously. We get it, you don't like GenAI. That's fine, but can we talk about it without getting normative?
- kjfaejgoaeijhei 1y agoThe article doesn't, but you can go to the "more information" section and keep digging: "To systematically assess differences between LLM-generated and human-written summaries, we also collected the corresponding expert-written summaries from NEJM Journal Watch (henceforth ‘NEJM JW’)"
- bethekidyouwant 1y agoRight so not scientific American how convenient (and may I add scientific!)
- ekianjo 1y ago> Chatbots were nearly five times more likely to produce broad generalisations than their human counterparts Never seen a journalist not make broad generalizations on a scientific study so I am not sure where these numbers are coming from.
- kjfaejgoaeijhei 1y agoYou can read the original research article here: https://royalsocietypublishing.org/doi/10.1098/rsos.241776 https://royalsocietypublishing.org/doi/10.1098/rsos.241776 It will tell you exactly where the numbers are coming from - the comparison is between (a variety of) LLMs and "expert-written summaries from NEJM Journal Watch".
- rekenaut 1y agoI wonder if there is any connection between the models producing exaggerated outputs and the litany of exaggerated or overconfident claims that academic media offices or the press have produced from previous studies. Maybe the models trained on the studies and the reports on the studies naturally tend toward the style of attention seeking reports even when directly provided with the studies.
- DoctorOW 1y agoThis is the same mistake we were seeing in commercial use of AI. 1. "This process is flawed due to human bias" 2. Train AI/ML to make the same decisions with the same outcome 3. "How can there be any flaws in this process? AI is bias-free."
- johnea 1y agoA repost of a previous comment, but the anthropomophization of this tech is so off the charts, I feel like I'm going to be repeating it, a lot: One of the most offensive words in the anthropomophization of LLMs is: hallucinate. It's not only an anthropomorphism, it's also a euphemism. A correct interpretation of the word would imply that the LLM has some fantastical vision that it mistakes for reality. What utter bullsh1t. Let's just use the correct word for this type of output: wrong. When the LLM generates a sequence of words, that may or may not be grammatically correct, but infers a state or conclusion that is not factually correct; lets state what actually happened: the LLM generated text was WRONG. It didn't take a trip down Alice's rabbit hole, it just put words together into a stream that inferred a piece of information that was incorrect, it was just WRONG. The euphemistic aspect of using this word is a greater offense than the anthropomorphism, because it's painting some cutesy picture of what happened, instead of accurately acknowledging that the s/w generated an incorrect result. It's covering up for the inherent short comings of the tech.
- dinfinity 1y agoImagine the irony if this article were to exaggerate the claims made in the study itself. Perhaps saying things like "Most leading chatbots routinely exaggerate science findings" instead of "We tested 10 prominent LLMs, including ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.3 70B, and Claude 3.7 Sonnet [...] with DeepSeek, ChatGPT-4o, and LLaMA 3.3 70B overgeneralizing in 26–73% of cases". To be fair, the article itself already mentions this: "Summaries by models (1), (4), (8), and (9) didn’t significantly differ in the kind of generalisations they contained from the original text. “So basically,” Peters concludes, “Claude, in different versions, did really well.”"
- GiorgioG 1y agoI am shocked, I tell you I’m flabbergasted that LLMs which cannot truly reason go down the wrong rabbit-hole nearly every time. I’ve spent a good chunk of the weekend trying to accelerate the development of a small SaaS solution using Cursor, CoPilot, etc even with the latest and greatest Claude Sonnet 4, paying for “Max”, etc and any meaningfully sized request, winds up being a completely frustrating experience of trying to get these tools to stay on the rails. I’m to the point where I will give these fuckers explicit instructions to come up with several hypotheses, potential solutions, and then not to generate code before getting my go-ahead and it still more often than not go ahead and start editing code without my approval/direction. As a bonus it will forget a few things we corrected earlier. Can’t wait for the first vibe-coder to get sued when there’s a massive breach or financial loss. These tools are not ready for prime time and they certainly aren’t worthy of the billions being spent on them.
- mannanj 1y agoI've been wondering this as well. did you use any prompts to help split into roles of product, engineering, well defined tasks.
- GiorgioG 1y agoI even used Claude to help me formulate my prompts, Cursor rules, etc.
- mannanj 1y agoI think were just training them.
- ninetyninenine 1y agoI'm sick of people saying LLMs don't reason as if they know unequivocally. We dont' understand what's going on with LLMs. We have evidence of them reasoning successfully. And we have evidence of them Failing to reason. That evidence DOES not logically lead to "LLMs don't reason". There are multiple possibilities here. 1. LLMs can reason, but they can't tell the difference between hallucination and reasoning. 2. LLMs can reason, but they choose to lie. 3. LLMs can't reason, when they get something right, it's pure coincidence. There is not a single person who can prove or disprove ANY of those 3 points. What most people end up doing is ironically Identical to what the LLM does. They hallucinate an answer: "LLMs cannot truly reason." Think about it. There is NO EVIDENCE or insight into how an LLM works that can even tell us how an LLM arrived at a specific response. We HAVE NOTHING. Yet why do I see everywhere people like this guy, who makes claims out of nowhere? Which brings us back full circle: do humans reason? How similar is human hallucination to LLM hallucination?
- deleted 1y ago[deleted]
- more_corn 1y agoAlso, most leading news outlets routinely exaggerate science findings
- Mistletoe 1y agoI’m sure they were trained on clickbait articles and PR releases from universities, which routinely do the same and misinterpret and overstate the importance of the science.
- proc0 1y agoStudies like this should make it evident that LLMs are not reasoning at all. An AI that would reason like humans would also make mistakes like humans and by now we can all see that LLM mistakes are completely random and nonsensical. It is also unclear that the current rate of progress is in the direction that would solve this issue. I think generative AI for images and video will get better, but the reasoning capabilities seem to be in a different domain.
- ants_everywhere 1y ago> Studies like this should make it evident that LLMs are not reasoning at all. An AI that would reason like humans.... Humans don't reason either. Reasoning is something we do in writing, especially with mathematical and logical notation. Just about everything else that feels like reasoning is something much less. This has been widely known at least since the stories where Socrates made everybody look like fools. But it's also what the psychological research shows. What people feel like they're doing when they're reasoning is very different with what they're actually doing.
- proc0 1y agoWell no, most people can reason without writing or speaking. I can just think and reason about anything. Not sure what you mean. Reasoning is something like structured thoughts. You have a series of thoughts that build on each other to produce some conclusion (also a thought). If we assume that the brain is a computer, then thoughts and reasoning are implemented on brain software with some kind of algorithm... and I think it's pretty obvious this algorithm is completely different than what happens in LLMs... to the extent that we can safely say it is not reasoning like the brain does. There is also a semantic argument here, if we say that since we don't know what humans are doing then we can also stretch the word and use it for AI, but I think this is muddying the waters and creating all the hype that I think will not deliver what it's promising.
- deleted 1y ago[deleted]
- 1y ago
- ekianjo 1y agoLLM being trained on a corpus of pitiful science reporting from the mainstream press... This is exactly what you would expect.
- Aurornis 1y agoOnline forums for difficult medical conditions are full of ChatGPT copy-and-paste responses right now. LLMs are the tool of choice for people who want answers and want them now. Much like this article claims, a popular usage is to prompt the LLM to get a more optimistic interpretation about a specific supplement or treatment. This is a really difficult problem because these people are often very sick and not getting the answers they want from their doctors. Previously this void was filled by alternative medicine doctors and quacks selling supplements. Now ChatGPT has arrived and has convinced a lot of them that they have a super-human AI at their fingertips that can be massaged to produce any answer they want to hear. It’s painful to try to read some of these forums where threads have turned into endless “here’s what ChatGPT says” pasted walls of text, followed by someone else trying to counter with a different ChatGPT wall of text. This isn’t unique to LLMs. There is a huge market for grossly exaggerating the conclusions of scientific studies. Podcasters like Huberman and Dr. Rhonda Patrick are famous for taking obscure studies with questionable conclusions and extrapolating to “protocols” or supplement stacks for their fans to follow. I often get downvoted when I mention fan-favorite podcasters by name, but I think by now many listeners have caught on to the way they exaggerate small studies into exciting listening material.
- kjfaejgoaeijhei 1y agoDefinitely interesting how LLMs seem to be very good at reinforcing confirmation bias...
- Aurornis 1y agoOne of my friends went down a path of using ChatGPT to validate his belief that his alternative medicine approach was better than what his doctors recommended. He had a whole host of tricks to work around ChatGPT’s protections or cautious replies. He’d strip out the cautions because he thought it was just OpenAI’s lawyers forcing disclaimers into the model. If he didn’t get the answer he wanted, he’d just retry or rephrase until he did. I think the other half of the problem is that LLM users can be very good at pushing the LLM into doing confirmation bias. It’s much easier when you can hit the retry button with no side effects, unlike a human who will recognize what you’re trying to do when you keep asking different variations of a question until you get the answer you want.
- delichon 1y ago“You asked her what color a house was and she said, ‘It’s white on this side.’” “That’s right.” “She didn’t assume that the other side was white, too… and a Fair Witness wouldn’t.” -- Stranger in a Strange Land (1961) An LLM is an abstraction machine, it mashes together anything that is nearby in a high dimensional space. Its statistical model is its source of truth. For a Fair Witness AI reasoning needs to supplant statistics. Which I'm guessing can get weird fast. LLMs are really good at being suggestible. For this we need the opposite.
- kjfaejgoaeijhei 1y agoThat's an extremely interesting observation, it kind of reminds me of some of the traps bayesians sometimes fall into... (not saying bayesian reasoning can't be very useful)