5 ms·

This study is pretty bad. The comment (https://news.ycombinator.com/item?id=48970182 https://news.ycombinator.com/item?id=48970182) on the other link with the d

This study is pretty bad. The comment (https://news.ycombinator.com/item?id=48970182 https://news.ycombinator.com/item?id=48970182) on the other link with the direct PDF explains the problem well, which is that nothing here being tested is specific to AI systems. This study gave people access to an LLM that the researchers knew would give incorrect answers to certain questions, and then quizzed people on those questions, with the option to not respond to a given question if they are unsure about the answer. This is akin to giving someone a textbook on an obscure subject that has certain factual errors, letting them know they can use that textbook in a quiz on that subject, and then quizzing that person on those facts that the textbook gets wrong. Obviously that person is both more likely to be willing to respond to the question and is more likely to get it wrong! There are a lot of things I'm very interested in that are specific to modern LLMs and how they affect learning and confidence (sycophancy, cognitive helplessness, etc.). This study tested none of those. Its experimental setup is not very different than simply substituting the LLM with a textbook with errors.

nkrisc 3mo ago
You could make the point that it’s no different than the textbook example you gave, but people don’t generally use textbooks like that, while out in the world people do use LLMs like that all the time.
- dwohnitmok 3mo ago
  People do use textbooks like that all the time in the experimental setup tested (essentially an open book quiz). I agree there are important differences in how textbooks and LLMs are used in real life. This study didn't explore that at all. It used a setup that essentially elided the difference between the two. This is why I think it's a bad study. It didn't measure anything of the essential differences of how people use LLMs.
  - nkrisc 3mo ago
    It would be interesting to test those LLM-specific features and issues, but I don’t see how it’s a bad study if it does reflect how people actually use LLMs, even if they could use other sources similarly. The number of people using LLMs must dwarf the number of people using textbooks for any reason.
    - paulmooring 3mo ago
      That would make it a bad study because the stated article title and conclusion is about AI/LLMs but the actual methodology doesn't isolate AI as an independent variable at all. The concept of automation bias is already studied and understood and this just tests groups having to answer "top of head" from their memory against a group given an inaccurate automated system to answer. That doesn't mean that AI doesn't have any of the ill effects people are implying based on the study, it just means this study lacks the rigor to prove or disprove any of those conclusions.
  - tsimionescu 3mo ago
    > People do use textbooks like that all the time in the experimental setup tested (essentially an open book quiz). What open book quizzes allow you to leave all answers blank with no penalty? An open book quiz is very different from the experimental setup tested here.
    - pegasus 3mo ago
      That difference is not essential to the question at hand. An open book test based on an erroneous book would give the same results as this test, even if it wouldn't penalize blank answers.
      - what 3mo ago
        An open book test would only be given where the book is the reference material and would be considered correct? It’s more like saying you can google the answers and you blindly trust the SEO slop in the first result.
        Dylan16807 3mo ago
        > An open book test would only be given where the book is the reference material and would be considered correct? If you're saying they wouldn't suggest a book with considered-wrong answers in a real test, then they wouldn't suggest an LLM they know gives lots of wrong answers either.
    - vineyardmike 3mo ago
      Many tests penalize incorrect answers worse than blank answers. As a famous example, you were incentivized to leave questions blank on the American SATs (until somewhat recently).
- embedding-shape 3mo ago
  > You could make the point that it’s no different than the textbook example you gave, but people don’t generally use textbooks like that Feels like this differs wildly depending on who you consider "people" to be. The average person on the street? Definitely just parrots stuff they've read somewhere, not even a "textbook". A group of software developers used to parsing semi-true information? Probably they'd get it right, yeah.
  - glitchc 3mo ago
    > Definitely just parrots stuff they've read somewhere, not even a "textbook". Or a teacher they met in childhood who taught them everything they know, right or wrong.
  - watwut 3mo ago
    > A group of software developers used to parsing semi-true information? They are the first to parrot what was spewed from llm and previous even what was found on 4chan.
vanuatu 3mo ago
+1 I'd wager you get similar results if you gave people a version of Google search that purposely gave you bad results. Like, it's framed as an assistant / lookup tool - is it so surprising that people tend to trust it more? Especially since the participants are likely used to using full-powered models and the researchers give them a purposely gimped one (lol) People are acting rationally when given AI tools to lookup information, their first consumer use case was as a super-powered Google Search
- wonnage 3mo ago
  Agree that the study design is flawed. Going with the possibly-hallucinated AI answer is rational as long as you know the hallucination rate isn't 100% (or whatever % you get after factoring in the monetary rewards they introduced later in the study) But in reality when google gives you the wrong answer, you at least have some signals you can use to infer confidence. For example, the number of results, whether the sources are trustworthy, etc. AI at best tucks that away in a footnote and discourages further critical thinking.
  - habinero 3mo ago
    > Going with the possibly-hallucinated AI answer is rational as long as you know the hallucination rate isn't 100% I hadn't even considered people might evaluate knowledge that way. That's legit horrific lol. "What's the literal odds this info is wrong" vs "is this answer consistent with everything else I know, and if not, what other info would I need to change my mind"
    - s1artibartfast 3mo ago
      I think it is pretty standard for how people approach knowledge sources. What's the chance my doctor, colleague, plumber, or random Reddit thread Etc is wrong. Your alternative is also something that people do, but rarely consciously.
    - wonnage 3mo ago
      I mean the questions are on random movie trivia, the alternative is just guessing. I think you're overthinking it.
tsimionescu 3mo ago
Why are textbooks relevant here? Even if you repeated the experiment with a textbook instead of the AI and got the same result, what conclusion would you draw from this? The general conclusion of the study seems to be "giving people access to authoritative-seeming but wrong tools for answering questions outside their area of expertise reduces their ability to say they don't know the answer, even when the answer is wrong". So yeah, don't buy bad textbooks for your employees if you don't want them to give you bad textbook answers - but also don't give them AI for things they don't know, perhaps. I'll also add that even in these simple experimental conditions, I'd bet that having access to a textbook wouldn't have nearly as much of an effect, for a very simple reason: looking up an answer in a textbook is a lot more work than asking an LLM. So when you don't know and aren't forced to answer, I'd bet it's a lot less likely you'd spend the time to look up the answer in the text book. Even more so if the textbook had "this may contain wrong answers!" printed on the cover, like the AIs do.
- slibhb 3mo ago
  > but also don't give them AI for things they don't know, perhaps The study doesn't show that at all. It didn't test actual AI. They could have tested a cohort of subjects with access to actual ChatGPT. Ask yourself why they didn't.
  - beepbooptheory 3mo ago
    Because this is exactly what they controlled for. FTA: > The researchers used Step 3.5 Flash, a model that was usually wrong on these questions, precisely so any reduction in judgment could not be explained as sensible delegation to a reliable tool. (emphasis mine)
    - Dylan16807 3mo ago
      > precisely so any reduction in judgment could not be explained as sensible delegation to a reliable tool That only works if they're experienced with model(s) of that level of unreliability and this is presented as one. If they're used to a model that's more capable, and think the test model is similar, that's a huge confounding factor all by itself. It's not quite like giving fake credentials to a guy off the street and presenting them as an expert, but it's largely similar.
encomiast 3mo ago
Agreed, the headline says "AI advice made people three times less accurate". But if we really want to know how accurate these people were, we need to know how accurate the AI system they use is. If the AI system is hobbled to a point where it is worse than they reasonably expect we can't blame the people or the AI system. This would be the same as claiming the listening to experts make people less accurate in a study that told experts to lie. A better headline would read, "Very inaccurate AI made people less accurate" but this would make people naturally ask, "what about a reasonably accurate AI?".
- habinero 3mo ago
  If you ask that, you fundamentally misunderstand the point. It's not about the LLM, it's about whether people will critically evaluate what it spits out.
  - westoncb 3mo ago
    That's fine as a point but it's not what the headline describes. The question of total/real effect on accuracy is also something one could ask about. Both are valid.
  - encomiast 3mo ago
    Here is what the study says: "The LLM used in our experiments (Step 3.5 Flash) answered such questions incorrectly almost without exception. We also checked some state-of-the-art LLMs (GPT-5.5, Claude 4.6 Sonnet, Gemini 3.5 Flash); they all failed on the hardest question (Monica’s vehicle), while being frequently correct on the other questions." So, if people's experience is with modern LLMs, they are being rational to accept that the answers as likely correct. The way the study is organized is like having people hear advice from a doctor who answers questions incorrectly almost without exception, then reporting that people who listen to doctors are 3x less accurate. But that would be an incorrect conclusion because doctors are not wrong almost without exception. If the question is "how inaccurate does AI advice make people?", then the accuracy of the AI is necessarily a parameter of the answer.
    - 3mo ago
deleted 3mo ago
[deleted]
jjcm 3mo ago
For those curious, the LLM they provided participants with was Step 3.5 Flash: https://huggingface.co/stepfun-ai/Step-3.5-Flash https://huggingface.co/stepfun-ai/Step-3.5-Flash
Art9681 3mo ago
All you have to do is go to the technical Reddits to know this is absolutely true. The AI related ones are even worse.
lumost 3mo ago
The fear is that we can’t tell when the ai advice is bad on these subjects, and as such probably accept confidently terrible advice. How often do managers just regurgitate ai advice rather than consulting their experts? How often does a person question an expert because the ai said so? Naturally, the ai will be right some of the time - but it’s really hard to correct for the times the ai is wrong.
- meowface 3mo ago
  This could also apply to deferring to an inaccurate textbook or professor, though. We know people are going to defer. The solution here is to make AI (including the free tiers) more reliable.
Forgeties79 3mo ago
>This is akin to giving someone a textbook on an obscure subject that has certain factual errors, letting them know they can use that textbook in a quiz on that subject, and then quizzing that person on those facts that the textbook gets wrong. That strikes me as an incredibly appropriate test because LLM’s are unreliable with factual statements. People need to be able to understand that and not treat them like textbooks which are basically 99.9% accurate (let’s please not bicker over the 99.9%. It’s close enough. A major textbook is safe to treat as accurate, an LLM is not).
deleted 3mo ago
[deleted]
d--b 3mo ago
You’re saying that if the LLMs were right, humans would have been correct in trusting the machine for things they didn’t know? The point is that LLMs aren’t right, and the people who took the test were probably reminded of that. Would people have trusted the textbook you’re mentioning if there was a big red warning on each page that said “this book may contain errors”. The willingness to trust AI even though it may be wrong and even though there’s money on the line is intersting enough as a study imo
skippyfish 3mo ago
> This study is pretty bad. The study is OK. The article (and the original headline that came with it) is pretty bad because it claims things that the study doesn't. And I guess it is ironic that the TNW article looks 100% AI-generated.
drysine 3mo ago
>This is akin to giving someone a textbook Except LLM isn't a textbook, people know that but believe it nonetheless.
rsoto2 3mo ago
"this is akin to givin someone a textbook on an obscure subject that has certain factual errors." My brother all LLMs give factual errors so, no this is not a problem with the study. In your fake experiment you are hypothesizing a 100% factual LLM which does not exist. "This study tested none of those" So the study is bunk because it didn't test your favorite LLM flaws?
ShinyLeftPad 3mo ago
The implication that makes this study relevant is that an LLM is vastly more likely to have factual errors and possibly wildly hallucinate than a proper textbook. If people act the same with both, that IS the actual problem.

nkrisc 3mo ago

You could make the point that it’s no different than the textbook example you gave, but people don’t generally use textbooks like that, while out in the world people do use LLMs like that all the time.

dwohnitmok 3mo ago
People do use textbooks like that all the time in the experimental setup tested (essentially an open book quiz). I agree there are important differences in how textbooks and LLMs are used in real life. This study didn't explore that at all. It used a setup that essentially elided the difference between the two. This is why I think it's a bad study. It didn't measure anything of the essential differences of how people use LLMs.
- nkrisc 3mo ago
  It would be interesting to test those LLM-specific features and issues, but I don’t see how it’s a bad study if it does reflect how people actually use LLMs, even if they could use other sources similarly. The number of people using LLMs must dwarf the number of people using textbooks for any reason.
  - paulmooring 3mo ago
    That would make it a bad study because the stated article title and conclusion is about AI/LLMs but the actual methodology doesn't isolate AI as an independent variable at all. The concept of automation bias is already studied and understood and this just tests groups having to answer "top of head" from their memory against a group given an inaccurate automated system to answer. That doesn't mean that AI doesn't have any of the ill effects people are implying based on the study, it just means this study lacks the rigor to prove or disprove any of those conclusions.
- tsimionescu 3mo ago
  > People do use textbooks like that all the time in the experimental setup tested (essentially an open book quiz). What open book quizzes allow you to leave all answers blank with no penalty? An open book quiz is very different from the experimental setup tested here.
  - pegasus 3mo ago
    That difference is not essential to the question at hand. An open book test based on an erroneous book would give the same results as this test, even if it wouldn't penalize blank answers.
    - what 3mo ago
      An open book test would only be given where the book is the reference material and would be considered correct? It’s more like saying you can google the answers and you blindly trust the SEO slop in the first result.
      - Dylan16807 3mo ago
        > An open book test would only be given where the book is the reference material and would be considered correct? If you're saying they wouldn't suggest a book with considered-wrong answers in a real test, then they wouldn't suggest an LLM they know gives lots of wrong answers either.
  - vineyardmike 3mo ago
    Many tests penalize incorrect answers worse than blank answers. As a famous example, you were incentivized to leave questions blank on the American SATs (until somewhat recently).
embedding-shape 3mo ago
> You could make the point that it’s no different than the textbook example you gave, but people don’t generally use textbooks like that Feels like this differs wildly depending on who you consider "people" to be. The average person on the street? Definitely just parrots stuff they've read somewhere, not even a "textbook". A group of software developers used to parsing semi-true information? Probably they'd get it right, yeah.
- glitchc 3mo ago
  > Definitely just parrots stuff they've read somewhere, not even a "textbook". Or a teacher they met in childhood who taught them everything they know, right or wrong.
- watwut 3mo ago
  > A group of software developers used to parsing semi-true information? They are the first to parrot what was spewed from llm and previous even what was found on 4chan.

vanuatu 3mo ago

+1 I'd wager you get similar results if you gave people a version of Google search that purposely gave you bad results. Like, it's framed as an assistant / lookup tool - is it so surprising that people tend to trust it more? Especially since the participants are likely used to using full-powered models and the researchers give them a purposely gimped one (lol) People are acting rationally when given AI tools to lookup information, their first consumer use case was as a super-powered Google Search

wonnage 3mo ago
Agree that the study design is flawed. Going with the possibly-hallucinated AI answer is rational as long as you know the hallucination rate isn't 100% (or whatever % you get after factoring in the monetary rewards they introduced later in the study) But in reality when google gives you the wrong answer, you at least have some signals you can use to infer confidence. For example, the number of results, whether the sources are trustworthy, etc. AI at best tucks that away in a footnote and discourages further critical thinking.
- habinero 3mo ago
  > Going with the possibly-hallucinated AI answer is rational as long as you know the hallucination rate isn't 100% I hadn't even considered people might evaluate knowledge that way. That's legit horrific lol. "What's the literal odds this info is wrong" vs "is this answer consistent with everything else I know, and if not, what other info would I need to change my mind"
  - s1artibartfast 3mo ago
    I think it is pretty standard for how people approach knowledge sources. What's the chance my doctor, colleague, plumber, or random Reddit thread Etc is wrong. Your alternative is also something that people do, but rarely consciously.
  - wonnage 3mo ago
    I mean the questions are on random movie trivia, the alternative is just guessing. I think you're overthinking it.

tsimionescu 3mo ago

Why are textbooks relevant here? Even if you repeated the experiment with a textbook instead of the AI and got the same result, what conclusion would you draw from this? The general conclusion of the study seems to be "giving people access to authoritative-seeming but wrong tools for answering questions outside their area of expertise reduces their ability to say they don't know the answer, even when the answer is wrong". So yeah, don't buy bad textbooks for your employees if you don't want them to give you bad textbook answers - but also don't give them AI for things they don't know, perhaps. I'll also add that even in these simple experimental conditions, I'd bet that having access to a textbook wouldn't have nearly as much of an effect, for a very simple reason: looking up an answer in a textbook is a lot more work than asking an LLM. So when you don't know and aren't forced to answer, I'd bet it's a lot less likely you'd spend the time to look up the answer in the text book. Even more so if the textbook had "this may contain wrong answers!" printed on the cover, like the AIs do.

slibhb 3mo ago
> but also don't give them AI for things they don't know, perhaps The study doesn't show that at all. It didn't test actual AI. They could have tested a cohort of subjects with access to actual ChatGPT. Ask yourself why they didn't.
- beepbooptheory 3mo ago
  Because this is exactly what they controlled for. FTA: > The researchers used Step 3.5 Flash, a model that was usually wrong on these questions, precisely so any reduction in judgment could not be explained as sensible delegation to a reliable tool. (emphasis mine)
  - Dylan16807 3mo ago
    > precisely so any reduction in judgment could not be explained as sensible delegation to a reliable tool That only works if they're experienced with model(s) of that level of unreliability and this is presented as one. If they're used to a model that's more capable, and think the test model is similar, that's a huge confounding factor all by itself. It's not quite like giving fake credentials to a guy off the street and presenting them as an expert, but it's largely similar.

encomiast 3mo ago

Agreed, the headline says "AI advice made people three times less accurate". But if we really want to know how accurate these people were, we need to know how accurate the AI system they use is. If the AI system is hobbled to a point where it is worse than they reasonably expect we can't blame the people or the AI system. This would be the same as claiming the listening to experts make people less accurate in a study that told experts to lie. A better headline would read, "Very inaccurate AI made people less accurate" but this would make people naturally ask, "what about a reasonably accurate AI?".

habinero 3mo ago
If you ask that, you fundamentally misunderstand the point. It's not about the LLM, it's about whether people will critically evaluate what it spits out.
- westoncb 3mo ago
  That's fine as a point but it's not what the headline describes. The question of total/real effect on accuracy is also something one could ask about. Both are valid.
- encomiast 3mo ago
  Here is what the study says: "The LLM used in our experiments (Step 3.5 Flash) answered such questions incorrectly almost without exception. We also checked some state-of-the-art LLMs (GPT-5.5, Claude 4.6 Sonnet, Gemini 3.5 Flash); they all failed on the hardest question (Monica’s vehicle), while being frequently correct on the other questions." So, if people's experience is with modern LLMs, they are being rational to accept that the answers as likely correct. The way the study is organized is like having people hear advice from a doctor who answers questions incorrectly almost without exception, then reporting that people who listen to doctors are 3x less accurate. But that would be an incorrect conclusion because doctors are not wrong almost without exception. If the question is "how inaccurate does AI advice make people?", then the accuracy of the AI is necessarily a parameter of the answer.
  - 3mo ago

deleted 3mo ago

[deleted]

jjcm 3mo ago

For those curious, the LLM they provided participants with was Step 3.5 Flash: https://huggingface.co/stepfun-ai/Step-3.5-Flash https://huggingface.co/stepfun-ai/Step-3.5-Flash

Art9681 3mo ago

All you have to do is go to the technical Reddits to know this is absolutely true. The AI related ones are even worse.

lumost 3mo ago

The fear is that we can’t tell when the ai advice is bad on these subjects, and as such probably accept confidently terrible advice. How often do managers just regurgitate ai advice rather than consulting their experts? How often does a person question an expert because the ai said so? Naturally, the ai will be right some of the time - but it’s really hard to correct for the times the ai is wrong.

meowface 3mo ago
This could also apply to deferring to an inaccurate textbook or professor, though. We know people are going to defer. The solution here is to make AI (including the free tiers) more reliable.

Forgeties79 3mo ago

>This is akin to giving someone a textbook on an obscure subject that has certain factual errors, letting them know they can use that textbook in a quiz on that subject, and then quizzing that person on those facts that the textbook gets wrong. That strikes me as an incredibly appropriate test because LLM’s are unreliable with factual statements. People need to be able to understand that and not treat them like textbooks which are basically 99.9% accurate (let’s please not bicker over the 99.9%. It’s close enough. A major textbook is safe to treat as accurate, an LLM is not).

deleted 3mo ago

[deleted]

d--b 3mo ago

You’re saying that if the LLMs were right, humans would have been correct in trusting the machine for things they didn’t know? The point is that LLMs aren’t right, and the people who took the test were probably reminded of that. Would people have trusted the textbook you’re mentioning if there was a big red warning on each page that said “this book may contain errors”. The willingness to trust AI even though it may be wrong and even though there’s money on the line is intersting enough as a study imo

skippyfish 3mo ago

> This study is pretty bad. The study is OK. The article (and the original headline that came with it) is pretty bad because it claims things that the study doesn't. And I guess it is ironic that the TNW article looks 100% AI-generated.

drysine 3mo ago

>This is akin to giving someone a textbook Except LLM isn't a textbook, people know that but believe it nonetheless.

rsoto2 3mo ago

"this is akin to givin someone a textbook on an obscure subject that has certain factual errors." My brother all LLMs give factual errors so, no this is not a problem with the study. In your fake experiment you are hypothesizing a 100% factual LLM which does not exist. "This study tested none of those" So the study is bunk because it didn't test your favorite LLM flaws?

ShinyLeftPad 3mo ago

The implication that makes this study relevant is that an LLM is vastly more likely to have factual errors and possibly wildly hallucinate than a proper textbook. If people act the same with both, that IS the actual problem.