12 ms·

LLMs hallucinate 100% of the time - hallucinating is what they Do. Often they’re hallucinating things that also occur in reality, but it sounds like the legal p

LLMs hallucinate 100% of the time - hallucinating is what they Do. Often they’re hallucinating things that also occur in reality, but it sounds like the legal products are only doing so 66-83% of the time.

soulofmischief 2y ago
Words are meaningless if we don't adhere to some common definition. "Hallucination" and "generation" are two separate concepts, and it is unproductive to conflate them.
- mycall 2y ago
  Indeed. Humans hallucinate under these definitions.
  - roughly 2y ago
    Which, indeed, we do. The current understanding of perception is that sense inputs serve to correct the brain's existing model of the world - you do not look at something and then perceive it, you generate an image of it and then use your vision to correct it.
  - kevindamm 2y ago
    Actual hallucination is quite eye-opening in part because of the realization that perception lenses all our senses _all the time_. So while I agree that it's over-broadening the definition of hallucination, I think it ironically appropriate too.
  - jedbrown 2y ago
    The point is that LLMs are never right for the right reason. Humans who understand the subject matter can make mistakes, but they are mistakes of a different nature. The issue reminds me of this from Terry Tao (LLMs being not-even pre-rigorous, but adept at forging the style of rigorous exposition): It is perhaps worth noting that mathematicians at all three of the above stages of mathematical development can still make formal mistakes in their mathematical writing. However, the nature of these mistakes tends to be rather different, depending on what stage one is at: 1. Mathematicians at the pre-rigorous stage of development often make formal errors because they are unable to understand how the rigorous mathematical formalism actually works, and are instead applying formal rules or heuristics blindly. It can often be quite difficult for such mathematicians to appreciate and correct these errors even when those errors are explicitly pointed out to them. 2. Mathematicians at the rigorous stage of development can still make formal errors because they have not yet perfected their formal understanding, or are unable to perform enough “sanity checks” against intuition or other rules of thumb to catch, say, a sign error, or a failure to correctly verify a crucial hypothesis in a tool. However, such errors can usually be detected (and often repaired) once they are pointed out to them. 3. Mathematicians at the post-rigorous stage of development are not infallible, and are still capable of making formal errors in their writing. But this is often because they no longer need the formalism in order to perform high-level mathematical reasoning, and are actually proceeding largely through intuition, which is then translated (possibly incorrectly) into formal mathematical language. The distinction between the three types of errors can lead to the phenomenon (which can often be quite puzzling to readers at earlier stages of mathematical development) of a mathematical argument by a post-rigorous mathematician which locally contains a number of typos and other formal errors, but is globally quite sound, with the local errors propagating for a while before being cancelled out by other local errors. (In contrast, when unchecked by a solid intuition, once an error is introduced in an argument by a pre-rigorous or rigorous mathematician, it is possible for the error to propagate out of control until one is left with complete nonsense at the end of the argument.) https://terrytao.wordpress.com/career-advice/theres-more-to-mathematics-than-rigour-and-proofs/ https://terrytao.wordpress.com/career-advice/theres-more-to-...
- jerf 2y ago
  Then maybe they shouldn't be. "Generating" a legal reference and "hallucinating" it are pretty much the same thing. Lawyers need AIs that "remember" legal references or are better capable of "looking them up" on demand. "Generation" is the completely wrong thing to do here. In fact it is arguably the worst possible thing, inasmuch as simply failing would actually be far preferable. I wouldn't expect a human to be able to just spit out all relevant case law, completely correctly with every last aspect of the citation accurate down to every last digit, based on solely their internal language model either.
  - bbor 2y ago
    Memory is generation; the memory palace metaphor is much less accurate IMO than ones that focus on Hebbian learning, learned (Pavlovian) responses, and/or, ironically enough, machine learning. The only difference between “remembering” a legal reference and “generating” one is that the former denotes a much higher level of confidence. Which, ofc, we all agree is an inescapably important metric — an acceptable failure rate for these models would be maybe 5%, MAX. LLMs used alone through a browser window, even with a half-assed “RAG” solution, are not enough IMO.
    - jerf 2y ago
      I think what you'd find in a real lawyer's office is that they don't "remember" or "generate" very many references at all. Those books in the stereotypical image of a lawyer's office weren't there just there to look lawyerly, and in modern times those computers are equipped with some expensive subscriptions to legal databases. This is not how legal references are created, not now, not in the past. Depending on an LLM (not "AI", LLM specfically) to do this is insane, stupid, and anyone selling an LLM-based product for this purpose ought to be sued for fraud... and that only coincidentally since they are providing a product to lawyers. Why is anyone even defending this? LLMs are objectively unsuited for this. LLMs are objectively failing at this. LLM architecture is obviously not a good choice for this task to anyone with even a basic understanding of them. But... who cares? All this means is that LLMs are not suited for this task. I wouldn't be banging on this except for the engineers running around here with stars in their eyes thinking that despite all the evidence they've finally found the tool that does everything. No. We haven't. They don't. This isn't an attack on the viability of AI as a whole or LLMs specifically, any more than pointing out that a hammer is not a good tool for cutting grass is an attack on hammers.
- jrm4 2y ago
  I disagree. It makes sense TO conflate them so that people can better understand what's going on under the hood. We need to "de-personalize" these things as much as possible so we don't make stupid mistakes surrounding them. Think of it like this: How USEFUL is it to say "I googled 'the closest ice cream shop to my house' but it 'hallucinated' the one that was the second-closest'" It's just a dumb term.
  - jonfw 2y ago
    > "I googled 'the closest ice cream shop to my house' but it 'hallucinated' the one that was the second-closest'" a google search doesn't do that though. It will actually select ice cream shops from it's database, determine their location, your location, and calculate the distance. And then select the closest. It may mis-calculate, but it will not "hallucinate". This is incredibly different than what LLMs do, which is why hallucinate is appropriate for an LLM but not appropriate for a conventional search or map engine
    - jrm4 2y ago
      I fail to see how this difference strongly justifies the word "hallucination." The computer is crunching numbers and probabilities of things and giving its best guess based on that, in both cases. If it's appropriate for one, it should be for the other. (which, I'd say, it isn't, because "hallucination" implies "mistake by a human-like thing")
      - jonfw 2y ago
        > The computer is crunching numbers and probabilities of things and giving its best guess based on that, in both cases. If it's appropriate for one, it should be for the other. Google maps isn't choosing a nearest ice cream store based on probability, it's using an algorithm designed for this purpose.
        jrm4 2y ago
        What do you percieve to be significant about this difference? I'm not saying there isn't a difference, but I fail to see why it matters.
mjburgess 2y ago
For the sake of sanity, I think we just have to acknowledge these are propagandistic terms that serve the interests of those who invent them, and are design to mystify and cofound non-experts. Then, move on. I don't really think there's any winning this propaganda war until it shatters under its own illusions. It's shouting into the wind.
neolefty 2y ago
Maybe you mean "work from an internal model of the world"?
IIAOPSW 2y ago
Fine, reality is a shared hallucination. I still want to know what fraction of the time it shares our hallucination.
bbor 2y ago
Everyone’s giving you grief when you’re clearly merely making the Kantian assertion that all experience must have both immediate and mediated elements, namely “our subjective perceptions” and “the unknowable ineffable objectively-undifferentiable Real World”. The best we can do is construct our little hallucinations and hope (and try to empirically ensure!) that they match whatever Reality might be, knowing that we’ll never have a true fundamental link to it. Well, it’s either that or you were being a human elitist and portraying a difference of degree as a difference in kind. Idk if we’ve settled on terms yet but I’d probably level ones like “automatist”, “bio supremacist”, or maybe “radical humanist”…
- roughly 2y ago
  > Everyone’s giving you grief when you’re clearly merely making the Kantian assertion that all experience must have both immediate and mediated elements, namely “our subjective perceptions” and “the unknowable ineffable objectively-undifferentiable Real World”. The best we can do is construct our little hallucinations and hope (and try to empirically ensure!) that they match whatever Reality might be, knowing that we’ll never have a true fundamental link to it. Insofar as the mechanism by which LLMs operate mirrors the mechanisms by which the brain operates - which they do, to some degree - yes, the fact that LLMs hallucinate reflect the gap between perceptions, thoughts, and "reality" in human behavior, and there is indeed a deeper philosophical lesson to be drawn here. If LLMs manage to disabuse the field of software engineering of its apparently common conceptions about the brain works and reflects reality, I'll consider it a win. > Well, it’s either that or you were being a human elitist and portraying a difference of degree as a difference in kind. Idk if we’ve settled on terms yet but I’d probably level ones like “automatist”, “bio supremacist”, or maybe “radical humanist”… I'm not sure how I could be considered a bio-supremacist for pointing out that a hammer is not a drill. LLMs are phenomenally complicated tools, but as currently deployed, they're tools, not beings. We may at some point create a system that one could credibly claim is an actual artificial intelligence, and that system may contain things that look or operate like LLMs, but right now what we're looking at are large statistical model that we're operating by providing it inputs and reading the outputs. A first-line criteria here, for me, would be that the system operates continuously (ie: not just when someone puts a token in the box). The LLMs we're discussing here do not do that. Someone generates a series of tokens, provides those as inputs to the equation, and then runs the equation, and then decides whether to continue to provide inputs into the equation until they're satisfied by the result. That's a tool, not a "being".
throw46365 2y ago
Exactly!
localfirst 2y ago
It's not that you can't build systems with low rate of success, its when you couple tightly to a blackbox controlled by a small number of people and prohibitively expensive to have any insight infrastructure if LLM was just another "framework" that a php developer could pick up we could build systems to deal with these "hallucinations".

soulofmischief 2y ago

Words are meaningless if we don't adhere to some common definition. "Hallucination" and "generation" are two separate concepts, and it is unproductive to conflate them.

mycall 2y ago
Indeed. Humans hallucinate under these definitions.
- roughly 2y ago
  Which, indeed, we do. The current understanding of perception is that sense inputs serve to correct the brain's existing model of the world - you do not look at something and then perceive it, you generate an image of it and then use your vision to correct it.
- kevindamm 2y ago
  Actual hallucination is quite eye-opening in part because of the realization that perception lenses all our senses _all the time_. So while I agree that it's over-broadening the definition of hallucination, I think it ironically appropriate too.
- jedbrown 2y ago
  The point is that LLMs are never right for the right reason. Humans who understand the subject matter can make mistakes, but they are mistakes of a different nature. The issue reminds me of this from Terry Tao (LLMs being not-even pre-rigorous, but adept at forging the style of rigorous exposition): It is perhaps worth noting that mathematicians at all three of the above stages of mathematical development can still make formal mistakes in their mathematical writing. However, the nature of these mistakes tends to be rather different, depending on what stage one is at: 1. Mathematicians at the pre-rigorous stage of development often make formal errors because they are unable to understand how the rigorous mathematical formalism actually works, and are instead applying formal rules or heuristics blindly. It can often be quite difficult for such mathematicians to appreciate and correct these errors even when those errors are explicitly pointed out to them. 2. Mathematicians at the rigorous stage of development can still make formal errors because they have not yet perfected their formal understanding, or are unable to perform enough “sanity checks” against intuition or other rules of thumb to catch, say, a sign error, or a failure to correctly verify a crucial hypothesis in a tool. However, such errors can usually be detected (and often repaired) once they are pointed out to them. 3. Mathematicians at the post-rigorous stage of development are not infallible, and are still capable of making formal errors in their writing. But this is often because they no longer need the formalism in order to perform high-level mathematical reasoning, and are actually proceeding largely through intuition, which is then translated (possibly incorrectly) into formal mathematical language. The distinction between the three types of errors can lead to the phenomenon (which can often be quite puzzling to readers at earlier stages of mathematical development) of a mathematical argument by a post-rigorous mathematician which locally contains a number of typos and other formal errors, but is globally quite sound, with the local errors propagating for a while before being cancelled out by other local errors. (In contrast, when unchecked by a solid intuition, once an error is introduced in an argument by a pre-rigorous or rigorous mathematician, it is possible for the error to propagate out of control until one is left with complete nonsense at the end of the argument.) https://terrytao.wordpress.com/career-advice/theres-more-to-mathematics-than-rigour-and-proofs/ https://terrytao.wordpress.com/career-advice/theres-more-to-...
jerf 2y ago
Then maybe they shouldn't be. "Generating" a legal reference and "hallucinating" it are pretty much the same thing. Lawyers need AIs that "remember" legal references or are better capable of "looking them up" on demand. "Generation" is the completely wrong thing to do here. In fact it is arguably the worst possible thing, inasmuch as simply failing would actually be far preferable. I wouldn't expect a human to be able to just spit out all relevant case law, completely correctly with every last aspect of the citation accurate down to every last digit, based on solely their internal language model either.
- bbor 2y ago
  Memory is generation; the memory palace metaphor is much less accurate IMO than ones that focus on Hebbian learning, learned (Pavlovian) responses, and/or, ironically enough, machine learning. The only difference between “remembering” a legal reference and “generating” one is that the former denotes a much higher level of confidence. Which, ofc, we all agree is an inescapably important metric — an acceptable failure rate for these models would be maybe 5%, MAX. LLMs used alone through a browser window, even with a half-assed “RAG” solution, are not enough IMO.
  - jerf 2y ago
    I think what you'd find in a real lawyer's office is that they don't "remember" or "generate" very many references at all. Those books in the stereotypical image of a lawyer's office weren't there just there to look lawyerly, and in modern times those computers are equipped with some expensive subscriptions to legal databases. This is not how legal references are created, not now, not in the past. Depending on an LLM (not "AI", LLM specfically) to do this is insane, stupid, and anyone selling an LLM-based product for this purpose ought to be sued for fraud... and that only coincidentally since they are providing a product to lawyers. Why is anyone even defending this? LLMs are objectively unsuited for this. LLMs are objectively failing at this. LLM architecture is obviously not a good choice for this task to anyone with even a basic understanding of them. But... who cares? All this means is that LLMs are not suited for this task. I wouldn't be banging on this except for the engineers running around here with stars in their eyes thinking that despite all the evidence they've finally found the tool that does everything. No. We haven't. They don't. This isn't an attack on the viability of AI as a whole or LLMs specifically, any more than pointing out that a hammer is not a good tool for cutting grass is an attack on hammers.
jrm4 2y ago
I disagree. It makes sense TO conflate them so that people can better understand what's going on under the hood. We need to "de-personalize" these things as much as possible so we don't make stupid mistakes surrounding them. Think of it like this: How USEFUL is it to say "I googled 'the closest ice cream shop to my house' but it 'hallucinated' the one that was the second-closest'" It's just a dumb term.
- jonfw 2y ago
  > "I googled 'the closest ice cream shop to my house' but it 'hallucinated' the one that was the second-closest'" a google search doesn't do that though. It will actually select ice cream shops from it's database, determine their location, your location, and calculate the distance. And then select the closest. It may mis-calculate, but it will not "hallucinate". This is incredibly different than what LLMs do, which is why hallucinate is appropriate for an LLM but not appropriate for a conventional search or map engine
  - jrm4 2y ago
    I fail to see how this difference strongly justifies the word "hallucination." The computer is crunching numbers and probabilities of things and giving its best guess based on that, in both cases. If it's appropriate for one, it should be for the other. (which, I'd say, it isn't, because "hallucination" implies "mistake by a human-like thing")
    - jonfw 2y ago
      > The computer is crunching numbers and probabilities of things and giving its best guess based on that, in both cases. If it's appropriate for one, it should be for the other. Google maps isn't choosing a nearest ice cream store based on probability, it's using an algorithm designed for this purpose.
      - jrm4 2y ago
        What do you percieve to be significant about this difference? I'm not saying there isn't a difference, but I fail to see why it matters.

mjburgess 2y ago

For the sake of sanity, I think we just have to acknowledge these are propagandistic terms that serve the interests of those who invent them, and are design to mystify and cofound non-experts. Then, move on. I don't really think there's any winning this propaganda war until it shatters under its own illusions. It's shouting into the wind.

neolefty 2y ago

Maybe you mean "work from an internal model of the world"?

IIAOPSW 2y ago

Fine, reality is a shared hallucination. I still want to know what fraction of the time it shares our hallucination.

bbor 2y ago

Everyone’s giving you grief when you’re clearly merely making the Kantian assertion that all experience must have both immediate and mediated elements, namely “our subjective perceptions” and “the unknowable ineffable objectively-undifferentiable Real World”. The best we can do is construct our little hallucinations and hope (and try to empirically ensure!) that they match whatever Reality might be, knowing that we’ll never have a true fundamental link to it. Well, it’s either that or you were being a human elitist and portraying a difference of degree as a difference in kind. Idk if we’ve settled on terms yet but I’d probably level ones like “automatist”, “bio supremacist”, or maybe “radical humanist”…

roughly 2y ago
> Everyone’s giving you grief when you’re clearly merely making the Kantian assertion that all experience must have both immediate and mediated elements, namely “our subjective perceptions” and “the unknowable ineffable objectively-undifferentiable Real World”. The best we can do is construct our little hallucinations and hope (and try to empirically ensure!) that they match whatever Reality might be, knowing that we’ll never have a true fundamental link to it. Insofar as the mechanism by which LLMs operate mirrors the mechanisms by which the brain operates - which they do, to some degree - yes, the fact that LLMs hallucinate reflect the gap between perceptions, thoughts, and "reality" in human behavior, and there is indeed a deeper philosophical lesson to be drawn here. If LLMs manage to disabuse the field of software engineering of its apparently common conceptions about the brain works and reflects reality, I'll consider it a win. > Well, it’s either that or you were being a human elitist and portraying a difference of degree as a difference in kind. Idk if we’ve settled on terms yet but I’d probably level ones like “automatist”, “bio supremacist”, or maybe “radical humanist”… I'm not sure how I could be considered a bio-supremacist for pointing out that a hammer is not a drill. LLMs are phenomenally complicated tools, but as currently deployed, they're tools, not beings. We may at some point create a system that one could credibly claim is an actual artificial intelligence, and that system may contain things that look or operate like LLMs, but right now what we're looking at are large statistical model that we're operating by providing it inputs and reading the outputs. A first-line criteria here, for me, would be that the system operates continuously (ie: not just when someone puts a token in the box). The LLMs we're discussing here do not do that. Someone generates a series of tokens, provides those as inputs to the equation, and then runs the equation, and then decides whether to continue to provide inputs into the equation until they're satisfied by the result. That's a tool, not a "being".

throw46365 2y ago

Exactly!

localfirst 2y ago

It's not that you can't build systems with low rate of success, its when you couple tightly to a blackbox controlled by a small number of people and prohibitively expensive to have any insight infrastructure if LLM was just another "framework" that a php developer could pick up we could build systems to deal with these "hallucinations".