3 ms·

> I think of them as similar to human optical illusions. What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity",

> I think of them as similar to human optical illusions. What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans than anything to do with what we refer to as "hallucinations" in humans—only they don't have the ability to analyze whether or not they're making up something or accurately parameterizing the vibes. The only connection between the two concepts is that the initial imagery from DeepDream was super trippy.

AdieuToLogic 2y ago
> What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... No. What people call LLM "hallucinations" is the result of a PRNG[0] influencing an algorithm to pursue a less statistically probable branch without regard nor understanding. 0 - https://en.wikipedia.org/wiki/Pseudorandom_number_generator https://en.wikipedia.org/wiki/Pseudorandom_number_generator
- majormajor 2y ago
  That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation. Consider the errors like "this math library will have this specific function" (based on a hundred other math libraries for other languages usually having that).
  - AdieuToLogic 2y ago
    > That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation. I believe we are saying the same thing here. My clarification to the OP's statement: What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... Was that the algorithm has no concept of correctness (nor the other anthropomorphic attributes cited), but instead relies on pseudo-randomness to vary search paths when generating text.
    - cornel_io 2y ago
      There are various results that suggest that LLMs do internally have everything they'd need to know that they're hallucinating/wrong: https://arxiv.org/abs/2402.09733 https://arxiv.org/abs/2402.09733 https://arxiv.org/abs/2305.18248 https://arxiv.org/abs/2305.18248 https://www.ox.ac.uk/news/2024-06-20-major-research-hallucinating-generative-models-advances-reliability-artificial https://www.ox.ac.uk/news/2024-06-20-major-research-hallucin... So I don't think it's that they have no concept of correctness, they do, but it's not strong enough. We're probably just not training them in ways that optimize for that over other desirable qualities, at least aggressively enough. It's also clear to anyone who has used many different models over the years that the amount of hallucination goes down as the models get better, even without any special attention being (apparently) paid to that problem. GPT 3.5 was REALLY bad about this stuff, but 4o and o1 are at least mediocre. So it may be that it's just one of the tougher things for a model to figure out, even if it's possible with massive capacity and compute. But I'd say it's very clear that we're not in the world Gary Marcus wishes we were in, where there's some hard and fundamental limitation that keeps a transformer network from having the capability to be more truthful as a it gets better; rather, like all aspects, we just aren't as far along as we'd prefer.
      - ForTheKidz 2y ago
        > There are various results that suggest that LLMs do internally have everything they'd need to know that they're hallucinating/wrong We need better definitions of what sort of reasonable expectation people can have for detecting incoherency and self-contradiction when humans are horrible at seeing this, except in comparison to things that don't seem to produce meaningful language in the general case. We all have contradictory worldviews and are therefore capable of rationally finding ourselves with conclusions that are trivially and empirically incoherent. I think "hallucinations" (horribly, horribly named term) are just an intractable burden of applying finite, lossy filters to a virtually continuous and infinitely detailed reality—language itself is sort of an ad-hoc, buggy consensus algorithm that's been sufficient to reproduce. But yea if you're looking for a coherent and satisfying answer on idk politics, values, basically anything that hinges on floating signifiers, you're going to have a bad time. (Or perhaps you're just hallucinating understanding and agreement: there are many phrases in the english language that read differently based on expected context and tone. It wouldn't surprise me if some models tended towards production of ambiguous or tautological semantics pleasingly-hedged or "responsibly"-moderated, aka PR.) Personally, I don't think it's a problem. If you are willing to believe what a chatbot says without verifying it there's little advice I could give you that can help. It's also good training to remind yourself that confidence is a poor signal for correctness.
- ForTheKidz 2y ago
  We really need an idiom for the behavior of being technically correct but absolutely destroying the prospect of interesting conversation. With this framing we might as well go back to arguing over which rock our local river god has blessed with greater utility. I'm not actually entirely convinced humans are capable of understanding much when discussion desired is this low quality. Critically, creation does not require intent nor understanding. Neither does recombination; neither reformulation. The only thing intent is necessary for is to create something meaningful to humans—handily taken care of via prompt and training material, just like with humans. (If you can't tell, I thought we had bypassed the neuroticism over whether or not data counts as "understanding", whatever that means to people, on week 2 of LLMs)
  - AdieuToLogic 2y ago
    > We really need an idiom for the behavior of being technically correct but absolutely destroying the prospect of interesting conversation. While it is not an idiom, the applicable term is likely pedantry[0]. > I'm not actually entirely convinced humans are capable of understanding much when discussion desired is this low quality. Ignoring the judgemental qualifier, consider your original post to which I replied: What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... The term for this behavior is anthropomorphism[1] due to ascribing human behaviors/motivations to algorithmic constructs. > Critically, creation does not require intent nor understanding. Neither does recombination; neither reformulation. The same can be said for a random number generator and a permutation algorithm. > (If you can't tell, I thought we had bypassed the neuroticism over whether or not data counts as "understanding", whatever that means to people, on week 2 of LLMs) If you can't tell, I differentiate between humans and algorithms, no matter the cleverness observed of the latter, as only the former can possess "understanding." 0 - https://www.merriam-webster.com/dictionary/pedant https://www.merriam-webster.com/dictionary/pedant 1 - https://www.merriam-webster.com/dictionary/anthropomorphism https://www.merriam-webster.com/dictionary/anthropomorphism
majormajor 2y ago
Inventiveness/creativity/imagination are deliberate things. LLM "hallucinations" are more akin to a student looking at a test over material they only 70% remember grabbing at what they think is the most likely correct answer. More "willful hope in the face of forgetting" than "creativity." Many LLM hallucinations - especially of the coding sort - are ones that would be obviously-wrong based on the training material, but the hundreds of languages/libraries/frameworks the thing was trained on start to blur together and there is not precise 100%-memorization recall but instead a "probably something like this" guess. It's not "inventive" to assume one math library will have the same functions as another, it's just losing sight of specific details.
- ForTheKidz 2y ago
  > Inventiveness/creativity/imagination are deliberate things. Not really. At least, it's just as much a reflex as any other human behavior to my perception. Anyway, why does intention—although I think this is mostly nonsensical/incoherent/a category error applied to LLMs—even matter to you? Either we have no goals and we're just idly discussing random word games (aka philosophy), which is fine with me, or we do have goals and whether or not you believe the software is intelligent or not is irrelevant. In the latter case anthropomorphizing discussion with words like "hallucination", "obviously", "deliberate", etc are just going to cause massive friction, distraction, and confusion. Why can't people be satisfied with "bad output"?
- TeMPOraL 2y ago
  > LLM "hallucinations" are more akin to a student looking at a test over material they only 70% remember grabbing at what they think is the most likely correct answer. AKA. extrapolation. AKA. what everyone is doing to a lesser or greater degree, when consequences of stopping are worse than of getting this wrong. That's not just the case of school, where giving up because you "don't know" is guaranteed F, while extrapolating has a non-zero chance of scoring you anything between F and A. It's also the case in everyday life, where you do things incrementally - getting the wrong answer is a stepping stone to getting a less wrong answer in the next attempt. We do that at every scale - from inner thought process all the way to large-scale engineering. Hardly anyone learns 100% of the material, because that's just plain memorization. We're always extrapolating from incomplete information; more studying and more experience (and more smarts) just makes us more likely to get it right. > It's not "inventive" to assume one math library will have the same functions as another, it's just losing sight of specific details. Depends. To a large extent, this kind of "hallucinations" is what a good programmer is supposed to be doing. That is, code to the API you'd like to have, inventing functions and classes convenient to you if they don't exist, and then see how to make this work - which, in one place, means fixing your own call sites, and in another, building utilities or a whole compat layer between your code and the actual API.
Applejinx 2y ago
If and only if the LLM is able to bring the novel, unexpected connection into itself and see whether it forms other consistent networks that lead to newly common associations and paths. A lot of us have had that experience. We use that ability to distinguish between 'genius thinkers' and 'kid overdosing on DMT'. It's not the ability to turn up the weird connections and go 'ooooh sparkly', it's whether you can build new associations that prove to be structurally sound. If that turns out to be something self-modifying large models (not necessarily 'language' models!) can do, that'll be important indeed. I don't see fiddling with the 'temperature' as the same thing, that's more like the DMT analogy. You can make the static model take a trip all you like, but if nothing changes nothing changes.
cratermoon 2y ago
the word you're looking for is "confabulation"

AdieuToLogic 2y ago

> What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... No. What people call LLM "hallucinations" is the result of a PRNG[0] influencing an algorithm to pursue a less statistically probable branch without regard nor understanding. 0 - https://en.wikipedia.org/wiki/Pseudorandom_number_generator https://en.wikipedia.org/wiki/Pseudorandom_number_generator

majormajor 2y ago
That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation. Consider the errors like "this math library will have this specific function" (based on a hundred other math libraries for other languages usually having that).
- AdieuToLogic 2y ago
  > That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation. I believe we are saying the same thing here. My clarification to the OP's statement: What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... Was that the algorithm has no concept of correctness (nor the other anthropomorphic attributes cited), but instead relies on pseudo-randomness to vary search paths when generating text.
  - cornel_io 2y ago
    There are various results that suggest that LLMs do internally have everything they'd need to know that they're hallucinating/wrong: https://arxiv.org/abs/2402.09733 https://arxiv.org/abs/2402.09733 https://arxiv.org/abs/2305.18248 https://arxiv.org/abs/2305.18248 https://www.ox.ac.uk/news/2024-06-20-major-research-hallucinating-generative-models-advances-reliability-artificial https://www.ox.ac.uk/news/2024-06-20-major-research-hallucin... So I don't think it's that they have no concept of correctness, they do, but it's not strong enough. We're probably just not training them in ways that optimize for that over other desirable qualities, at least aggressively enough. It's also clear to anyone who has used many different models over the years that the amount of hallucination goes down as the models get better, even without any special attention being (apparently) paid to that problem. GPT 3.5 was REALLY bad about this stuff, but 4o and o1 are at least mediocre. So it may be that it's just one of the tougher things for a model to figure out, even if it's possible with massive capacity and compute. But I'd say it's very clear that we're not in the world Gary Marcus wishes we were in, where there's some hard and fundamental limitation that keeps a transformer network from having the capability to be more truthful as a it gets better; rather, like all aspects, we just aren't as far along as we'd prefer.
    - ForTheKidz 2y ago
      > There are various results that suggest that LLMs do internally have everything they'd need to know that they're hallucinating/wrong We need better definitions of what sort of reasonable expectation people can have for detecting incoherency and self-contradiction when humans are horrible at seeing this, except in comparison to things that don't seem to produce meaningful language in the general case. We all have contradictory worldviews and are therefore capable of rationally finding ourselves with conclusions that are trivially and empirically incoherent. I think "hallucinations" (horribly, horribly named term) are just an intractable burden of applying finite, lossy filters to a virtually continuous and infinitely detailed reality—language itself is sort of an ad-hoc, buggy consensus algorithm that's been sufficient to reproduce. But yea if you're looking for a coherent and satisfying answer on idk politics, values, basically anything that hinges on floating signifiers, you're going to have a bad time. (Or perhaps you're just hallucinating understanding and agreement: there are many phrases in the english language that read differently based on expected context and tone. It wouldn't surprise me if some models tended towards production of ambiguous or tautological semantics pleasingly-hedged or "responsibly"-moderated, aka PR.) Personally, I don't think it's a problem. If you are willing to believe what a chatbot says without verifying it there's little advice I could give you that can help. It's also good training to remind yourself that confidence is a poor signal for correctness.
ForTheKidz 2y ago
We really need an idiom for the behavior of being technically correct but absolutely destroying the prospect of interesting conversation. With this framing we might as well go back to arguing over which rock our local river god has blessed with greater utility. I'm not actually entirely convinced humans are capable of understanding much when discussion desired is this low quality. Critically, creation does not require intent nor understanding. Neither does recombination; neither reformulation. The only thing intent is necessary for is to create something meaningful to humans—handily taken care of via prompt and training material, just like with humans. (If you can't tell, I thought we had bypassed the neuroticism over whether or not data counts as "understanding", whatever that means to people, on week 2 of LLMs)
- AdieuToLogic 2y ago
  > We really need an idiom for the behavior of being technically correct but absolutely destroying the prospect of interesting conversation. While it is not an idiom, the applicable term is likely pedantry[0]. > I'm not actually entirely convinced humans are capable of understanding much when discussion desired is this low quality. Ignoring the judgemental qualifier, consider your original post to which I replied: What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... The term for this behavior is anthropomorphism[1] due to ascribing human behaviors/motivations to algorithmic constructs. > Critically, creation does not require intent nor understanding. Neither does recombination; neither reformulation. The same can be said for a random number generator and a permutation algorithm. > (If you can't tell, I thought we had bypassed the neuroticism over whether or not data counts as "understanding", whatever that means to people, on week 2 of LLMs) If you can't tell, I differentiate between humans and algorithms, no matter the cleverness observed of the latter, as only the former can possess "understanding." 0 - https://www.merriam-webster.com/dictionary/pedant https://www.merriam-webster.com/dictionary/pedant 1 - https://www.merriam-webster.com/dictionary/anthropomorphism https://www.merriam-webster.com/dictionary/anthropomorphism

majormajor 2y ago

Inventiveness/creativity/imagination are deliberate things. LLM "hallucinations" are more akin to a student looking at a test over material they only 70% remember grabbing at what they think is the most likely correct answer. More "willful hope in the face of forgetting" than "creativity." Many LLM hallucinations - especially of the coding sort - are ones that would be obviously-wrong based on the training material, but the hundreds of languages/libraries/frameworks the thing was trained on start to blur together and there is not precise 100%-memorization recall but instead a "probably something like this" guess. It's not "inventive" to assume one math library will have the same functions as another, it's just losing sight of specific details.

ForTheKidz 2y ago
> Inventiveness/creativity/imagination are deliberate things. Not really. At least, it's just as much a reflex as any other human behavior to my perception. Anyway, why does intention—although I think this is mostly nonsensical/incoherent/a category error applied to LLMs—even matter to you? Either we have no goals and we're just idly discussing random word games (aka philosophy), which is fine with me, or we do have goals and whether or not you believe the software is intelligent or not is irrelevant. In the latter case anthropomorphizing discussion with words like "hallucination", "obviously", "deliberate", etc are just going to cause massive friction, distraction, and confusion. Why can't people be satisfied with "bad output"?
TeMPOraL 2y ago
> LLM "hallucinations" are more akin to a student looking at a test over material they only 70% remember grabbing at what they think is the most likely correct answer. AKA. extrapolation. AKA. what everyone is doing to a lesser or greater degree, when consequences of stopping are worse than of getting this wrong. That's not just the case of school, where giving up because you "don't know" is guaranteed F, while extrapolating has a non-zero chance of scoring you anything between F and A. It's also the case in everyday life, where you do things incrementally - getting the wrong answer is a stepping stone to getting a less wrong answer in the next attempt. We do that at every scale - from inner thought process all the way to large-scale engineering. Hardly anyone learns 100% of the material, because that's just plain memorization. We're always extrapolating from incomplete information; more studying and more experience (and more smarts) just makes us more likely to get it right. > It's not "inventive" to assume one math library will have the same functions as another, it's just losing sight of specific details. Depends. To a large extent, this kind of "hallucinations" is what a good programmer is supposed to be doing. That is, code to the API you'd like to have, inventing functions and classes convenient to you if they don't exist, and then see how to make this work - which, in one place, means fixing your own call sites, and in another, building utilities or a whole compat layer between your code and the actual API.

Applejinx 2y ago

If and only if the LLM is able to bring the novel, unexpected connection into itself and see whether it forms other consistent networks that lead to newly common associations and paths. A lot of us have had that experience. We use that ability to distinguish between 'genius thinkers' and 'kid overdosing on DMT'. It's not the ability to turn up the weird connections and go 'ooooh sparkly', it's whether you can build new associations that prove to be structurally sound. If that turns out to be something self-modifying large models (not necessarily 'language' models!) can do, that'll be important indeed. I don't see fiddling with the 'temperature' as the same thing, that's more like the DMT analogy. You can make the static model take a trip all you like, but if nothing changes nothing changes.

cratermoon 2y ago

the word you're looking for is "confabulation"