7 ms·

> While the hallucination problem in LLMs is inevitable [0], they can be significantly reduced... Every article on hallucinations needs to start with this fact

> While the hallucination problem in LLMs is inevitable [0], they can be significantly reduced... Every article on hallucinations needs to start with this fact until we've hammered that into every "AI Engineer"'s head. Hallucinations are not a bug—they're not a different mode of operation, they're not a logic error. They're not even really a distinct kind of output. What they are is a value judgement we assign to the output of an LLM program. A "hallucination" is just output from an LLM-based workflow that is not fit for purpose. This means that all techniques for managing hallucinations (such as the ones described in TFA, which are good) are better understood as techniques for constraining and validating the probabilistic output of an LLM to ensure fitness for purpose—it's a process of quality control, and it should be approached as such. The trouble is that we software engineers have spent so long working in an artificially deterministic world that we're not used to designing and evaluating probabilistic quality control systems for computer output. [0] They link to this paper: https://arxiv.org/pdf/2401.11817 https://arxiv.org/pdf/2401.11817

swatcoder 2y ago
> The trouble is that we software engineers have spent so long working in an artificially deterministic world that we're not used to designing and evaluating probabilistic quality control systems for computer output. I think that's a mischaracterization and not really accurate. As a trade, we're familiar with probabilistic/non-deterministic components and how to approach them. You were closer when you used quotes around "AI Engineer" -- many of the loudest people involved in generative AI right now have little to no grounding in engineering at all. They aren't used to looking at their work through "fit for purpose" concerns, compromises, efficiency, limits, constraints, etc -- whether that work uses AI or not. The rest of us are variously either working quietly, getting drowned out, or patiently waiting for our respected colleagues-in-engineering to document, demonstrate, and mature these very promising tools for us. Everything else you said is 100% right, though.
- deleted 2y ago
  [deleted]
- bongodongobob 2y ago
  You're missing his point. He's saying if you make a program, you expect it to do X reliably. X may include "send an email, or kick off this workflow, or add this to the log, or crash" but you don't expect it to, for example, "delete system32 and shut down the computer". LLMs have essentially unconstrained outputs where the above mentioned program couldn't possibly delete anything or shut down your computer because nothing even close to that is in the code. Please do not confuse this example with agentic AI losing the plot, that's not what I'm trying to say. Edit: a better example is that when you build an autocomplete plugin for your email client, you don't expect it to also be able to play chess. But look what happened.
  - gerdesj 2y ago
    [flagged]
    - deleted 2y ago
      [deleted]
- JKCalhoun 2y ago
  > As a trade, we're familiar with probabilistic/non-deterministic components and how to approach them. Yes, users.
  - mdp2021 2y ago
    And that small or large subsets of occasional or consistent bad reasoners we may have sometimes called "users" (in the secrecy of the four walls) reinforced, by contrast and by forcing us to look at things objectively trying to understand their "rants", the idea of proper reasonable stance, did it not?
  - mycall 2y ago
    ..and bugs, especially with analog computers.
- Workaccount2 2y ago
  Take all computers and make it so all memory has a 0.1-5% chance of bit flipping any second (depending on cost and temperature). That this just became a fundamental truth of reality. Any bit, anywhere in memory. It would completely turn SWE work on it's head. This is kind of how traditional engineering is, since reality is analog and everything is on a spectrum interacting with everything else all the time. There is no simple function where you put in 1 and get out 0. Everything in reality is put in 1 +/- .25 and get out 0 +/- .25. It's the reason why the complexity of hardware is trivial compared to the complexity of software.
  - swatcoder 2y ago
    That's not really engaging with the point because you're suggesting turning all of our tools into something grossly unreliable. Of course that's a radical shift from what anybody's used to and undermines every practice in the trade. But your mistake is just reinforcing what I wrote, because its the same mistake that the "loud people" are make when they think about generative AI. They imagine it as being a wholesale replacement for how projects are implemented and even how they're built in the first place. But the many experienced engineers looking at generative AI recognize it as one of many tools that they can turn to while building a project that fulfills their requirements. And like all their tools, it has capabilities, costs, and limitations that need to be considered. That its sometimes non-deterministic is not a new kind of cost or limitation. It's a challenging one, but not a novel one, and one just mindfully (or analytically) considers whether and how that non-determinism can be leveraged, minimized, etc. That is engineering, and it's what many of us have been doing with all sorts of tools for decades.
    - Workaccount2 2y ago
      Perhaps I am not explaining this well. What you call grossly unreliable and a radical shift from what any [SWE] is used to, is called Tuesday afternoon for a mechanical, electrical, civil, chemical, etc. etc. engineer. Call them classic engineers. Statistical outputs are the only outputs of classical engineering. You have never in your life assigned x = 5 and then later queried it and gotten x = 4.83. But that happens all the time in classic engineering, to the point that it is classic engineering. That's what the OP is trying to get across. LLM's are statistical systems that need statistical management. SWE's don't deal with statistical systems because like you said: >[statistical software systems would be] turning all of our tools into something grossly unreliable. Of course that's a radical shift from what anybody's used to and undermines every practice in the trade. Which is exactly why OP is saying SWE's need a new approach here.
leprechaun1066 2y ago
Calling them hallucinations was a huge mistake.
- sfink 2y ago
  The term is actually fine. The problem is when it's divorced from the reality of: > in some sense, hallucination is all LLMs do. They are dream machines. If you understand that, then the term "hallucination" makes perfect sense. Note that this in no way invalidates your point, because the term is constantly used and understood without this context. We would have avoided a lot of confusion if we had based it on the phrase "make shit up" and called it "shit" from the start. Marketing trumps accuracy again... (Also note that I am not using shit in a pejorative sense here. Making shit up is exactly what they're for, and what we want them to do. They come up with a lot of really good shit.)
  - jebarker 2y ago
    I agree with your point, but I don't think anthropomorphizing LLMs is helpful. They're statistical estimators trained by curve fitting. All generations are equally valid for the training data, objective and architecture. To me it's much clearer to think about it that way versus crude analogies to human brains.
    - threeseed 2y ago
      We can't expect end users to understand what "statistical estimators trained by curve fitting" means. That's why we use high level terms like hallucination. Because it's something everyone can understand even if it's not completely accurate.
      - jebarker 2y ago
        That's a good point. But re: not anthropomorphizing, what's wrong with errors, mistakes or inaccuracies? That's something everybody is familiar with and is more accurate. I'd guess most people have never actually experienced a hallucination anyway, so we're appealing to some vague notion of what that is.
zmgsabst 2y ago
A challenge is that it’s not easy to limit hallucinations without also limiting imagination and synthesis. In humans. But also apparently in LLMs.
- namaria 2y ago
  > A challenge is that it’s not easy to limit hallucinations without also limiting imagination and synthesis. > In humans. True, but distinguishing reality from imagination is a cornerstone of mental health. And it's becoming apparent that the average person will take the confident spurious affirmations of LLMs as facts, which should call their mental health into question.
  - zmgsabst 2y ago
    Misremembering facts isn’t a negative mental health event, yet is an example of imagination rather than recall — similar to LLMs hallucinating. Humans imagine events all the time, without the ability to know that happened. Part of why eye-witness testimony is so unreliable.
- mort96 2y ago
  Healthy humans generally have some internal model of the world against which they can judge what they're about to say. They can introspect and determine whether what they say is a guess or a statement of fact. LLMs can't.
  - zmgsabst 2y ago
    Humans routinely misremember facts but are relatively certain those remembrances are correct. That’s a form of minor, everyday hallucination. If you engage in such thorough criticism and checking of every recalled fact as to eliminate that, you’ll crush your ability to synthesize or compose new content.
    - mort96 2y ago
      No, that's not hallucination. In a human, there is a distinction between "this is information I truly think I know, my intention is to state a true fact about the world" and "this is something I don't know so I made something up". That distinction doesn't exist in LLMs. The fact that humans can be mistaken is a completely different issue.
tengbretson 2y ago
LLMs outputs are no more "hallucinations" than my output would be if I were asked to judge a dressage competition.
- xienze 2y ago
  I’ve had multiple occasions where I’ve asked an LLM how to do <whatever> in Java and it’ll very confidently answer to use <some class in some package that doesn’t exist>. It would be far more helpful to me to receive an answer like “I don’t think there’s a third party library that does this, you’ll have to write it yourself” than to waste my time telling me a lie. If anything, calling these outputs “hallucinations” is a very polite way of saying that the LLM is bullshitting the user.
  - tengbretson 2y ago
    LLMs don't know things, they just string together responses that are a best fit for what follows from their prompt. I suspect its so hard to get them to say "I don't know" because if they were biased towards responding that way then I would assume thats almost all they would ever say, since "I don't know" is an appropriate answer to every question imaginable.
    - JKCalhoun 2y ago
      I get that, but since it is all probabilities, you might imagine even the LLM knows when it is skating on thin ice. If I'm beginning with "Once / upon / a" I think the data will show a very high confidence in the word to follow with. So too I would imagine it would know when the trail of breadcrumbs it has been following is of the trashier and low probability kind. So just tell me. (Or perhaps speak to me and when your confidence is low you can drift into vocal fry territory.)
      - LPisGood 2y ago
        [dead]
      - 2y ago
rationalfaith 2y ago
[dead]
wpietri 2y ago
Excellent point: > just output from an LLM-based workflow that is not fit for purpose And I think this is just one aspect of what I think of as the stone soup [1] problem. Outside of rigorous test conditions, humans just have a hard time telling how much work they're doing when they interpret something. It's the same sort of thing you see with "psychics" doing things like cold reading. People make meaning out of vaguery and nonsense and then credit the nonsense-producer with the work. [1] https://en.wikipedia.org/wiki/Stone_Soup https://en.wikipedia.org/wiki/Stone_Soup
LgLasagnaModel 2y ago
‘ I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines. ...’ https://x.com/karpathy/status/1733299213503787018 https://x.com/karpathy/status/1733299213503787018
Animats 2y ago
> While the hallucination problem in LLMs is inevitable Oh, please. That's the same old computability argument used to claim that program verification is impossible. Computability isn't the problem. LLMs are forced to a reply, regardless of the quality of the reply. If "Confidence level is too low for a reply" is an option, the argument in that paper becomes invalid. The trouble is that we don't know how to get a confidence metric out of an LLM. This is the underlying problem behind hallucinations. As I've said before, if somebody doesn't crack that problem soon, the AI industry is overvalued. Alibaba's QwQ [1] supposedly is better at reporting when it doesn't know something. Comments on that? This article is really an ad for Kapa, which seems to offer managed AI as a service, or something like that. They hang various checkers and accessories on an LLM to try to catch bogus outputs. That's a patch, not a fix. [1] https://techcrunch.com/2024/11/27/alibaba-releases-an-open-challenger-to-openais-o1-reasoning-model/ https://techcrunch.com/2024/11/27/alibaba-releases-an-open-c...
- mort96 2y ago
  Confidence levels aren't necessarily low for incorrect replies, that's the problem. The LLM doesn't "know" that what it's outputting is incorrect. It just knows that the words it's writing are probable given the inputs; "this is how answers tend to look like". You can make improvements, as your parent comment already said, but it's not a problem which can be solved, only to some degree be reduced.
- lolinder 2y ago
  > Computability isn't the problem. LLMs are forced to a reply, regardless of the quality of the reply. If "Confidence level is too low for a reply" is an option, the argument in that paper becomes invalid. This is false. The confidence level of these models does not encode facts, it encodes statistical probabilities that a particular word would be the next one in the training data set. One source of output that is not fit for purpose (i.e. hallucinations) is unfit information in the training data, which is a problem that's intractable given the size of the data required to train a base model. You can reduce this problem by managing your training data better, but that's not possible to do perfectly, which gets to my point—managing hallucinations is entirely about risk management and reducing probabilities of failure to an acceptable level. It's not decidable, it's only manageable, and that only for applications that are low enough stakes that a 99.9% (or whatever) success rate is acceptable. It's a quality control problem, and one that will always be a battle. > Alibaba's QwQ [1] supposedly is better at reporting when it doesn't know something. Comments on that? I've been trying it out, and what it's actually better at is going in circles indefinitely, giving the illusion of careful thought. This can possibly be useful, but it's just as likely to "hallucinate" reasons why its first (correct) response might have been wrong (reasons that make no sense) as it is to correctly correct itself.
mdp2021 2y ago
> is inevitable False. It is (in this context) outputting a partial before full processing. Adequate (further) processing removes that "inevitable". Current architectures are not "final". Proper process: "It seems like that." // "Is it though?" // "Actually it isn't." (Edit: already this post had to be corrected many times because of errors...)
russnes 2y ago
so what you're saying is that LLMs are like middle aged men, just throwing things out there seeing if they'll stick?
emil_sorensen 2y ago
That's a great point. Reminds me of the "feature, not a bug" Karpathy tweet [0]. [0]: https://x.com/karpathy/status/1733299213503787018?lang=en https://x.com/karpathy/status/1733299213503787018?lang=en
- mike_hearn 2y ago
  ... which is linked to from the article ;) He's right but do people really misunderstand this? I think it's pretty clear that the issue is one of over-creativity. The hallucination problem is IMHO at heart two things that the fine article itself doesn't touch on: 1. The training sets contain few examples of people expressing uncertainty because the social convention on the internet is that if you don't know the answer, you don't post. Children also lie like crazy for the same reason, they ask simple questions so rarely see examples of their parents expressing uncertainty or refusing to answer, and it then has to be explicitly trained out of them. Arguably that training often fails and lots of adults "hallucinate" a lot more than anyone is comfortable acknowledging. The evidence for this is that models do seem to know their own level of certainty pretty well, which is why simple tricks like saying "don't make things up" can actually work. There's some interesting interpretability work that also shows this, which is alluded to in the article as well. 2. We train one-size-fits all models but use cases vary a lot in how much "creativity" is allowed. If you're a customer help desk worker then the creativity allowed is practically zero, and the ideal worker from an executive's perspective is basically just a search engine and human voice over an interactive flowchart. In fact that's often all they are. But then we use the same models for creative writing, research, coding, summarization and other tasks that benefit from a lot of creative choices. That makes it very hard to teach the model how much leeway it has to be over-confident. For instance during coding a long reply that contains a few hallucinated utility methods is way more useful than a response of "I am not 100% certain I can complete that request correctly" but if you're asking questions of the form "does this product I use have feature X" then a hallucination could be terrible. Obviously, the compressive nature of LLMs means they can never eliminate hallucinations entirely, but we're so far from reaching any kind of theoretical limit here. Techniques like better RAG are practical solutions that work for now, but in the longer run I think we'll see different instruct-trained models trained for different positions on the creativity/confidence spectrum. Models already differ quite a bit. I use Claude for writing code but GPT-4o for answering coding related questions, because I noticed that ChatGPT is much less prone to hallucinations than Claude is. This may even become part of the enterprise offerings of model companies. Consumers get the creative chatbots that'll play D&D with them, enterprises get the disciplined rule followers that can be trusted to answer support tickets.
Guthur 2y ago
What are you taking about, is not artificially deterministic, it is like that by design. We are fortunate that we can use a logic to encode logic and have it for the most part so the same thing given a fix set of antecedents. We even want this in the "real" world, when I turn the wheel left on my car I don't want it turn left only when it feels like it, when that happens we rightly classify it as a failure. We have the tools to build very complex deterministic systems but for the most part we chose not to use them, because they hard or not familiar or whatever other excuse you might come up with.
- skydhash 2y ago
  The fact is that while the tools exist and may be easy to learn, there’s always that nebulous part called creativity, taste, craftsmanship, expertise, or whatever (like designing good software) that’s hard to acquire. Generative AI is good at giving the illusion you can have that part (by piggybacking on the work of everyone). Which is why people get upset when you shatter that illusion by saying that they still don’t have it.
GuB-42 2y ago
Of course they are a bug. Just that hallucination emerge from the normal function of a LLM doesn't make it "not a bug". No programmer in their right mind will call the lack of bound checking resulting in garbled output "not a bug", even though it is a totally normal thing to do from the point of view of a CPU. It is a bug and you need additional code to fix it, for example by checking for out-of-bounds condition and returning an error if it happens. Same thing for LLM hallucinations. LLMs naturally hallucinate, but it is not what we want, so it is a bug. And to fix it, we need to engineer solutions that prevent the hallucinations from happening, maybe resulting in an "I don't know" response that would be analogous to an error message. How you do it may be different from a simple "if", with probabilities and all that, but the general idea is the same: recognizing error cases and responding accordingly. I guess it is comes down to how you define a bug, but how else would you call a result that is not fit for purpose?
- devmor 2y ago
  A bug is generally treated as undefined and undesirable side effects of a program. Hallucinations are undesirable but not undefined. We know that the process creates them and expect them. It’d be like using floats to calculate dollars and cents and calling the resulting math a bug - it’s not, you just used the technology wrong.
- lolinder 2y ago
  A bug is defined as an unexpected defect. You can fix an unexpected defect by correcting the error in the code that led to the defect. In your example of lack of bounds checking there's a very concrete answer that will instantly fix the defect—add bounds checking. Hallucinations are not unexpected in LLMs and cannot be fixed by correcting an error in the code. Instead they are fundamental property of the computing paradigm that was chosen, one that has to be worked around. It's closer to network lag than it is to bounds checking—it's an undesirable characteristic, but one that we knew about when we chose to make a network application. We'll do our best to mitigate it to acceptable levels, but it's certainly not a bug, it's just a fact of the paradigm.
  - 9rx 2y ago
cainxinth 2y ago
> What they are is a value judgement we assign to the output of an LLM program. A "hallucination" is just output from an LLM-based workflow that is not fit for purpose. In other words, hallucinations are to LLMs what weeds are to plants.
smartmic 2y ago
A machine with probabilistic output generation cannot tell what is a hallucination and what is not. It does not know the difference between truth and everything else. It is us humans on the receiving end who have to classify the content - and that is the problem. We have little patience, time, or energy to do this verification work for every piece of information. That's why we have the human trait of trust, which has been at the core of human progress from the beginning. Now the question can be rephrased. Is it possible to trust AI information generators - what's to be done to build trust? And here is the difficulty - I do not know why I should ever trust a probabilistic system as long as it has this property and does not turn into a deterministic version of itself. I won't lower my standards for trusting people, for good reasons. But I cannot even raise the bar for trust in a machine above zero as long as it is driven by randomness at its core.
mycall 2y ago
Does it help discourse at all by instead calling hallucinations a negative perceived form of imagination?
drcwpl 2y ago
You are right - "Hallucinations are not a bug"

swatcoder 2y ago

> The trouble is that we software engineers have spent so long working in an artificially deterministic world that we're not used to designing and evaluating probabilistic quality control systems for computer output. I think that's a mischaracterization and not really accurate. As a trade, we're familiar with probabilistic/non-deterministic components and how to approach them. You were closer when you used quotes around "AI Engineer" -- many of the loudest people involved in generative AI right now have little to no grounding in engineering at all. They aren't used to looking at their work through "fit for purpose" concerns, compromises, efficiency, limits, constraints, etc -- whether that work uses AI or not. The rest of us are variously either working quietly, getting drowned out, or patiently waiting for our respected colleagues-in-engineering to document, demonstrate, and mature these very promising tools for us. Everything else you said is 100% right, though.

deleted 2y ago
[deleted]
bongodongobob 2y ago
You're missing his point. He's saying if you make a program, you expect it to do X reliably. X may include "send an email, or kick off this workflow, or add this to the log, or crash" but you don't expect it to, for example, "delete system32 and shut down the computer". LLMs have essentially unconstrained outputs where the above mentioned program couldn't possibly delete anything or shut down your computer because nothing even close to that is in the code. Please do not confuse this example with agentic AI losing the plot, that's not what I'm trying to say. Edit: a better example is that when you build an autocomplete plugin for your email client, you don't expect it to also be able to play chess. But look what happened.
- gerdesj 2y ago
  [flagged]
  - deleted 2y ago
    [deleted]
JKCalhoun 2y ago
> As a trade, we're familiar with probabilistic/non-deterministic components and how to approach them. Yes, users.
- mdp2021 2y ago
  And that small or large subsets of occasional or consistent bad reasoners we may have sometimes called "users" (in the secrecy of the four walls) reinforced, by contrast and by forcing us to look at things objectively trying to understand their "rants", the idea of proper reasonable stance, did it not?
- mycall 2y ago
  ..and bugs, especially with analog computers.
Workaccount2 2y ago
Take all computers and make it so all memory has a 0.1-5% chance of bit flipping any second (depending on cost and temperature). That this just became a fundamental truth of reality. Any bit, anywhere in memory. It would completely turn SWE work on it's head. This is kind of how traditional engineering is, since reality is analog and everything is on a spectrum interacting with everything else all the time. There is no simple function where you put in 1 and get out 0. Everything in reality is put in 1 +/- .25 and get out 0 +/- .25. It's the reason why the complexity of hardware is trivial compared to the complexity of software.
- swatcoder 2y ago
  That's not really engaging with the point because you're suggesting turning all of our tools into something grossly unreliable. Of course that's a radical shift from what anybody's used to and undermines every practice in the trade. But your mistake is just reinforcing what I wrote, because its the same mistake that the "loud people" are make when they think about generative AI. They imagine it as being a wholesale replacement for how projects are implemented and even how they're built in the first place. But the many experienced engineers looking at generative AI recognize it as one of many tools that they can turn to while building a project that fulfills their requirements. And like all their tools, it has capabilities, costs, and limitations that need to be considered. That its sometimes non-deterministic is not a new kind of cost or limitation. It's a challenging one, but not a novel one, and one just mindfully (or analytically) considers whether and how that non-determinism can be leveraged, minimized, etc. That is engineering, and it's what many of us have been doing with all sorts of tools for decades.
  - Workaccount2 2y ago
    Perhaps I am not explaining this well. What you call grossly unreliable and a radical shift from what any [SWE] is used to, is called Tuesday afternoon for a mechanical, electrical, civil, chemical, etc. etc. engineer. Call them classic engineers. Statistical outputs are the only outputs of classical engineering. You have never in your life assigned x = 5 and then later queried it and gotten x = 4.83. But that happens all the time in classic engineering, to the point that it is classic engineering. That's what the OP is trying to get across. LLM's are statistical systems that need statistical management. SWE's don't deal with statistical systems because like you said: >[statistical software systems would be] turning all of our tools into something grossly unreliable. Of course that's a radical shift from what anybody's used to and undermines every practice in the trade. Which is exactly why OP is saying SWE's need a new approach here.

leprechaun1066 2y ago

Calling them hallucinations was a huge mistake.

sfink 2y ago
The term is actually fine. The problem is when it's divorced from the reality of: > in some sense, hallucination is all LLMs do. They are dream machines. If you understand that, then the term "hallucination" makes perfect sense. Note that this in no way invalidates your point, because the term is constantly used and understood without this context. We would have avoided a lot of confusion if we had based it on the phrase "make shit up" and called it "shit" from the start. Marketing trumps accuracy again... (Also note that I am not using shit in a pejorative sense here. Making shit up is exactly what they're for, and what we want them to do. They come up with a lot of really good shit.)
- jebarker 2y ago
  I agree with your point, but I don't think anthropomorphizing LLMs is helpful. They're statistical estimators trained by curve fitting. All generations are equally valid for the training data, objective and architecture. To me it's much clearer to think about it that way versus crude analogies to human brains.
  - threeseed 2y ago
    We can't expect end users to understand what "statistical estimators trained by curve fitting" means. That's why we use high level terms like hallucination. Because it's something everyone can understand even if it's not completely accurate.
    - jebarker 2y ago
      That's a good point. But re: not anthropomorphizing, what's wrong with errors, mistakes or inaccuracies? That's something everybody is familiar with and is more accurate. I'd guess most people have never actually experienced a hallucination anyway, so we're appealing to some vague notion of what that is.

zmgsabst 2y ago

A challenge is that it’s not easy to limit hallucinations without also limiting imagination and synthesis. In humans. But also apparently in LLMs.

namaria 2y ago
> A challenge is that it’s not easy to limit hallucinations without also limiting imagination and synthesis. > In humans. True, but distinguishing reality from imagination is a cornerstone of mental health. And it's becoming apparent that the average person will take the confident spurious affirmations of LLMs as facts, which should call their mental health into question.
- zmgsabst 2y ago
  Misremembering facts isn’t a negative mental health event, yet is an example of imagination rather than recall — similar to LLMs hallucinating. Humans imagine events all the time, without the ability to know that happened. Part of why eye-witness testimony is so unreliable.
mort96 2y ago
Healthy humans generally have some internal model of the world against which they can judge what they're about to say. They can introspect and determine whether what they say is a guess or a statement of fact. LLMs can't.
- zmgsabst 2y ago
  Humans routinely misremember facts but are relatively certain those remembrances are correct. That’s a form of minor, everyday hallucination. If you engage in such thorough criticism and checking of every recalled fact as to eliminate that, you’ll crush your ability to synthesize or compose new content.
  - mort96 2y ago
    No, that's not hallucination. In a human, there is a distinction between "this is information I truly think I know, my intention is to state a true fact about the world" and "this is something I don't know so I made something up". That distinction doesn't exist in LLMs. The fact that humans can be mistaken is a completely different issue.

tengbretson 2y ago

LLMs outputs are no more "hallucinations" than my output would be if I were asked to judge a dressage competition.

xienze 2y ago
I’ve had multiple occasions where I’ve asked an LLM how to do <whatever> in Java and it’ll very confidently answer to use <some class in some package that doesn’t exist>. It would be far more helpful to me to receive an answer like “I don’t think there’s a third party library that does this, you’ll have to write it yourself” than to waste my time telling me a lie. If anything, calling these outputs “hallucinations” is a very polite way of saying that the LLM is bullshitting the user.
- tengbretson 2y ago
  LLMs don't know things, they just string together responses that are a best fit for what follows from their prompt. I suspect its so hard to get them to say "I don't know" because if they were biased towards responding that way then I would assume thats almost all they would ever say, since "I don't know" is an appropriate answer to every question imaginable.
  - JKCalhoun 2y ago
    I get that, but since it is all probabilities, you might imagine even the LLM knows when it is skating on thin ice. If I'm beginning with "Once / upon / a" I think the data will show a very high confidence in the word to follow with. So too I would imagine it would know when the trail of breadcrumbs it has been following is of the trashier and low probability kind. So just tell me. (Or perhaps speak to me and when your confidence is low you can drift into vocal fry territory.)
    - LPisGood 2y ago
      [dead]
    - 2y ago

rationalfaith 2y ago

[dead]

wpietri 2y ago

Excellent point: > just output from an LLM-based workflow that is not fit for purpose And I think this is just one aspect of what I think of as the stone soup [1] problem. Outside of rigorous test conditions, humans just have a hard time telling how much work they're doing when they interpret something. It's the same sort of thing you see with "psychics" doing things like cold reading. People make meaning out of vaguery and nonsense and then credit the nonsense-producer with the work. [1] https://en.wikipedia.org/wiki/Stone_Soup https://en.wikipedia.org/wiki/Stone_Soup

LgLasagnaModel 2y ago

‘ I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines. ...’ https://x.com/karpathy/status/1733299213503787018 https://x.com/karpathy/status/1733299213503787018

Animats 2y ago

> While the hallucination problem in LLMs is inevitable Oh, please. That's the same old computability argument used to claim that program verification is impossible. Computability isn't the problem. LLMs are forced to a reply, regardless of the quality of the reply. If "Confidence level is too low for a reply" is an option, the argument in that paper becomes invalid. The trouble is that we don't know how to get a confidence metric out of an LLM. This is the underlying problem behind hallucinations. As I've said before, if somebody doesn't crack that problem soon, the AI industry is overvalued. Alibaba's QwQ [1] supposedly is better at reporting when it doesn't know something. Comments on that? This article is really an ad for Kapa, which seems to offer managed AI as a service, or something like that. They hang various checkers and accessories on an LLM to try to catch bogus outputs. That's a patch, not a fix. [1] https://techcrunch.com/2024/11/27/alibaba-releases-an-open-challenger-to-openais-o1-reasoning-model/ https://techcrunch.com/2024/11/27/alibaba-releases-an-open-c...

mort96 2y ago
Confidence levels aren't necessarily low for incorrect replies, that's the problem. The LLM doesn't "know" that what it's outputting is incorrect. It just knows that the words it's writing are probable given the inputs; "this is how answers tend to look like". You can make improvements, as your parent comment already said, but it's not a problem which can be solved, only to some degree be reduced.
lolinder 2y ago
> Computability isn't the problem. LLMs are forced to a reply, regardless of the quality of the reply. If "Confidence level is too low for a reply" is an option, the argument in that paper becomes invalid. This is false. The confidence level of these models does not encode facts, it encodes statistical probabilities that a particular word would be the next one in the training data set. One source of output that is not fit for purpose (i.e. hallucinations) is unfit information in the training data, which is a problem that's intractable given the size of the data required to train a base model. You can reduce this problem by managing your training data better, but that's not possible to do perfectly, which gets to my point—managing hallucinations is entirely about risk management and reducing probabilities of failure to an acceptable level. It's not decidable, it's only manageable, and that only for applications that are low enough stakes that a 99.9% (or whatever) success rate is acceptable. It's a quality control problem, and one that will always be a battle. > Alibaba's QwQ [1] supposedly is better at reporting when it doesn't know something. Comments on that? I've been trying it out, and what it's actually better at is going in circles indefinitely, giving the illusion of careful thought. This can possibly be useful, but it's just as likely to "hallucinate" reasons why its first (correct) response might have been wrong (reasons that make no sense) as it is to correctly correct itself.

mdp2021 2y ago

> is inevitable False. It is (in this context) outputting a partial before full processing. Adequate (further) processing removes that "inevitable". Current architectures are not "final". Proper process: "It seems like that." // "Is it though?" // "Actually it isn't." (Edit: already this post had to be corrected many times because of errors...)

russnes 2y ago

so what you're saying is that LLMs are like middle aged men, just throwing things out there seeing if they'll stick?

emil_sorensen 2y ago

That's a great point. Reminds me of the "feature, not a bug" Karpathy tweet [0]. [0]: https://x.com/karpathy/status/1733299213503787018?lang=en https://x.com/karpathy/status/1733299213503787018?lang=en

mike_hearn 2y ago
... which is linked to from the article ;) He's right but do people really misunderstand this? I think it's pretty clear that the issue is one of over-creativity. The hallucination problem is IMHO at heart two things that the fine article itself doesn't touch on: 1. The training sets contain few examples of people expressing uncertainty because the social convention on the internet is that if you don't know the answer, you don't post. Children also lie like crazy for the same reason, they ask simple questions so rarely see examples of their parents expressing uncertainty or refusing to answer, and it then has to be explicitly trained out of them. Arguably that training often fails and lots of adults "hallucinate" a lot more than anyone is comfortable acknowledging. The evidence for this is that models do seem to know their own level of certainty pretty well, which is why simple tricks like saying "don't make things up" can actually work. There's some interesting interpretability work that also shows this, which is alluded to in the article as well. 2. We train one-size-fits all models but use cases vary a lot in how much "creativity" is allowed. If you're a customer help desk worker then the creativity allowed is practically zero, and the ideal worker from an executive's perspective is basically just a search engine and human voice over an interactive flowchart. In fact that's often all they are. But then we use the same models for creative writing, research, coding, summarization and other tasks that benefit from a lot of creative choices. That makes it very hard to teach the model how much leeway it has to be over-confident. For instance during coding a long reply that contains a few hallucinated utility methods is way more useful than a response of "I am not 100% certain I can complete that request correctly" but if you're asking questions of the form "does this product I use have feature X" then a hallucination could be terrible. Obviously, the compressive nature of LLMs means they can never eliminate hallucinations entirely, but we're so far from reaching any kind of theoretical limit here. Techniques like better RAG are practical solutions that work for now, but in the longer run I think we'll see different instruct-trained models trained for different positions on the creativity/confidence spectrum. Models already differ quite a bit. I use Claude for writing code but GPT-4o for answering coding related questions, because I noticed that ChatGPT is much less prone to hallucinations than Claude is. This may even become part of the enterprise offerings of model companies. Consumers get the creative chatbots that'll play D&D with them, enterprises get the disciplined rule followers that can be trusted to answer support tickets.

Guthur 2y ago

What are you taking about, is not artificially deterministic, it is like that by design. We are fortunate that we can use a logic to encode logic and have it for the most part so the same thing given a fix set of antecedents. We even want this in the "real" world, when I turn the wheel left on my car I don't want it turn left only when it feels like it, when that happens we rightly classify it as a failure. We have the tools to build very complex deterministic systems but for the most part we chose not to use them, because they hard or not familiar or whatever other excuse you might come up with.

skydhash 2y ago
The fact is that while the tools exist and may be easy to learn, there’s always that nebulous part called creativity, taste, craftsmanship, expertise, or whatever (like designing good software) that’s hard to acquire. Generative AI is good at giving the illusion you can have that part (by piggybacking on the work of everyone). Which is why people get upset when you shatter that illusion by saying that they still don’t have it.

GuB-42 2y ago

Of course they are a bug. Just that hallucination emerge from the normal function of a LLM doesn't make it "not a bug". No programmer in their right mind will call the lack of bound checking resulting in garbled output "not a bug", even though it is a totally normal thing to do from the point of view of a CPU. It is a bug and you need additional code to fix it, for example by checking for out-of-bounds condition and returning an error if it happens. Same thing for LLM hallucinations. LLMs naturally hallucinate, but it is not what we want, so it is a bug. And to fix it, we need to engineer solutions that prevent the hallucinations from happening, maybe resulting in an "I don't know" response that would be analogous to an error message. How you do it may be different from a simple "if", with probabilities and all that, but the general idea is the same: recognizing error cases and responding accordingly. I guess it is comes down to how you define a bug, but how else would you call a result that is not fit for purpose?

devmor 2y ago
A bug is generally treated as undefined and undesirable side effects of a program. Hallucinations are undesirable but not undefined. We know that the process creates them and expect them. It’d be like using floats to calculate dollars and cents and calling the resulting math a bug - it’s not, you just used the technology wrong.
lolinder 2y ago
A bug is defined as an unexpected defect. You can fix an unexpected defect by correcting the error in the code that led to the defect. In your example of lack of bounds checking there's a very concrete answer that will instantly fix the defect—add bounds checking. Hallucinations are not unexpected in LLMs and cannot be fixed by correcting an error in the code. Instead they are fundamental property of the computing paradigm that was chosen, one that has to be worked around. It's closer to network lag than it is to bounds checking—it's an undesirable characteristic, but one that we knew about when we chose to make a network application. We'll do our best to mitigate it to acceptable levels, but it's certainly not a bug, it's just a fact of the paradigm.
- 9rx 2y ago

cainxinth 2y ago

> What they are is a value judgement we assign to the output of an LLM program. A "hallucination" is just output from an LLM-based workflow that is not fit for purpose. In other words, hallucinations are to LLMs what weeds are to plants.

smartmic 2y ago

A machine with probabilistic output generation cannot tell what is a hallucination and what is not. It does not know the difference between truth and everything else. It is us humans on the receiving end who have to classify the content - and that is the problem. We have little patience, time, or energy to do this verification work for every piece of information. That's why we have the human trait of trust, which has been at the core of human progress from the beginning. Now the question can be rephrased. Is it possible to trust AI information generators - what's to be done to build trust? And here is the difficulty - I do not know why I should ever trust a probabilistic system as long as it has this property and does not turn into a deterministic version of itself. I won't lower my standards for trusting people, for good reasons. But I cannot even raise the bar for trust in a machine above zero as long as it is driven by randomness at its core.

mycall 2y ago

Does it help discourse at all by instead calling hallucinations a negative perceived form of imagination?

drcwpl 2y ago

You are right - "Hallucinations are not a bug"