8 ms·

This is, I think, a better way to think about LLM mistakes compared to the usual "hallucinations". I think of them as similar to human optical illusions. There

This is, I think, a better way to think about LLM mistakes compared to the usual "hallucinations". I think of them as similar to human optical illusions. There are things about the human visual cortex (and also other sensory systems, see the McGurk Effect [0]), that, when presented with certain kinds of inputs, will consistently produce wrong interpretations/outputs. Even when we are 100% ware of the issue, we can't prevent our brains from generating the incorrect interpretation. LLMs seem to have similar issues along dramatically different axes, axes that humans are not used to seeing these kinds of mistakes; where nearly no human would make this kind of mistake and so we interpret it (in my opinion incorrectly) as lack of ability or intelligence. Because these are engineered systems, we may figure out ways to solve these problems (although I personally think the best we will ever do is decrease their prevalence), but more important is probably learning to recognize the places that LLMs are likely to make these errors, and, as your comment suggests, design work flows and systems that can deal with them. [0] https://youtu.be/2k8fHR9jKVM https://youtu.be/2k8fHR9jKVM

admiralrohan 2y ago
Hallucinating is fine but overconfidence is the problem. But I heard it's not an easy problem to solve.
- jacksnipe 2y ago
  Unfortunately, in the system most of us work in today, I think overconfidence is an intelligent behavior.
  - onemoresoop 2y ago
    I find it extremely dumb to see overconfident people that really have nothing special about them or are even incompetent. These people are not contributing positively to the system, quite on the contrary.
    - xwolfi 2y ago
      But we don't work for the system, fundamentally, we work for ourselves, and the system incentivizes us to work for it by aligning our constraints: if you work that direction, you'll get that reward. Overconfident people ofc do not contribute positively to the system, but they skew the system reward's calculation towards them: I swear I've done that work in that direction, where's my reward ? In a sense, they are extremely successful: they managed to do very low effort, get very high reward, help themselves like all of us but at a much better profit margin, by sacrificing a system that, let's be honest, none of us care about really. Your problem maybe, is that you swallowed the little BS the system fed you while incentivizing you: that the system matters more than yourself, at least at a greater extent than healthy ? And you see the same thing with AI: these things convince people so deeply of their intelligence that it blew to such proportion that NVidia is now worth trillions. I had a colleague mumbling yesterday that his wife now speaks more with ChatGPT than him. Overconfidence is a positive attribute... for oneself.
      - onemoresoop 2y ago
        Not talking about THE system or any system in particular same way gaming the system doesn’t refer to any system but just cheating in general. And if you like overconfident people good for you, I can’t stand them because they’re hollow with no basis in reality, flawed like everyone, just pumping out their egos with hot air. And your reasoning that overconfidence is a positive attribute doesn’t make much sense to me but we’re entitled to our own opinions.
      - butlike 2y ago
        Overconfident people are conquerors. Conquerors do not contribute positively to a harmonious system, true, but I'm not so sure we can glean the system is supposed to be harmonious. If one contributes "positively" to the system, everyone's value increases and the solution becomes more homogenized. Once the system is homogenized enough, it becomes vulnerable to adversity from an outside force. If the system is not harmonious/non-homoginized, the attacker would be drawn to the most powerful point in the system. Overconfident people aren't evil, they're simply stressing the system to make sure it can handle adversity from an outside force. They're saying: "listen, I'm going to take what you have, and you should be so happy that's all I'm taking." So I think overconfidence is a positive attribute for the system as well as for the overconfident individual. It's not a positive attribute for the local parties getting run over by the overconfident individual.
- Terr_ 2y ago
  > overconfidence is the problem. The problem is a bit deeper than that, because what we perceive as "confidence" is itself also an illusion. The (real) algorithm takes documents and makes them longer, and some humans configured a document that looks like a conversation between "User" and "AssistantBot", and they also wrote some code to act-out things that look like dialogue for one of the characters. The (real) trait of confidence involves next-token statistics. In contrast, the character named AssistantBot is "overconfident" in exactly the same sense that a character named Count Dracula is "immortal", "brooding", or "fearful" of garlic, crucifixes, and sunlight. Fictional traits we perceive on fictional characters from reading text. Yes, we can set up a script where the narrator periodically re-describes AssistantBot as careful and cautious, and that might help a bit with stopping humans from over-trusting the story they are being read. But trying to ensure logical conclusions arise from cautious reasoning is... well, indirect at best, much like trying to make it better at math by narrating "AssistantBot was good at math and diligent at checking the numbers." > Hallucinating P.S.: "Hallucinations" and prompt-injection are non-ironic examples of "it's not a bug, it's a feature". There's no minor magic incantation that'll permanently banish them without damaging how it all works.
  - plaguuuuuu 2y ago
    I'd love to know if the conversational training set includes documents where the AI throws its hands up and goes "actually I have no idea". I'm guessing not.
    - Terr_ 2y ago
      There's also the problem of whether the LLM would learn to generate stories where the AssistantBot gives up in cases that match our own logical reasons, versus ones where the AssistantBot gives up because that's simply what AssistantBots in training-stories usually do when the User character uses words of disagreement and disapproval.
- patates 2y ago
  Hallucinating is a confidence problem, no? Say, they should be 100% confident that "0.3" follows "0.2 + 0.1 =", but a lot of floating point examples on the internet make them less confident. On a much more nuanced problem, "0.30000000000000004" may get more and more confidence. This is what makes them "hallucinate", did I get it wrong? (in other words, am I hallucinating myself? :) )
pydry 2y ago
I dunno hallucinations seem like a pretty human type of mistake to me. when i try to remember something my brain often synthesizes new things by filling in the gaps. This would be where I often say "i might be imagining it, but..." or "i could have sworn there was a..." In such cases the thing that saves the human brain is double checking against reality (e.g. googling it to make sure). Miscounting the number of r's in strawberry by glancing at the word also seems like a pretty human mistake.
- gitaarik 2y ago
  But it's different kinds of hallucinations. AI doesn't have a base understanding of how physics work. So they think it's acceptible if in a video some element on the background in a next frame might appear in front of another element that is on the foreground. So it's always necessary to keep correcting LLMs, because they only learn by example, and you can't express any possible outcome of any physical process just by example, because physical processes can be in infinate variations. LLMs can keep getting closer to match our physical reality, but when you zoom into the details you'll always find that it comes short. So you can never really trust an LLM. If we want to make an AI that doesn't make errors, it should understand how physics works.
  - pydry 2y ago
    I dont think the errors really are all that different. Ever since GPT-3.5 came out Ive been thinking that the errors were ones a human could have made under a similar context. >LLMs can keep getting closer to match our physical reality, but when you zoom into the details you'll always find that it comes short. Like humans. >So you can never really trust an LLM. Cant really trust a human either. That's why we set up elaborate human systems (science, checks and balances in government, law, freedom of speech, markets) to mitigate our constant tendency to be complete fuck ups. We hallucinate science that does not exist, lies to maintain our worldview, jump to conclusions about guilt, build businesses based upon bad beliefs, etc. >If we want to make an AI that doesn't make errors, it should understand how physics works An AI that doesnt make errors wouldnt be AGI it would be a godlike superintelligence. I dont think thats even feasible. I think a propensity to make errors is intrinsic to how intelligence functions. Physics is just one domain that they work in and Im pretty sure some of them already do have varying understandings of physics.
j45 2y ago
Some of the errors are caused by humans. Say, due to changing the chat to only pay attention to recent messages and not the middle, omitting critical details.
ForTheKidz 2y ago
> I think of them as similar to human optical illusions. What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans than anything to do with what we refer to as "hallucinations" in humans—only they don't have the ability to analyze whether or not they're making up something or accurately parameterizing the vibes. The only connection between the two concepts is that the initial imagery from DeepDream was super trippy.
- AdieuToLogic 2y ago
  > What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... No. What people call LLM "hallucinations" is the result of a PRNG[0] influencing an algorithm to pursue a less statistically probable branch without regard nor understanding. 0 - https://en.wikipedia.org/wiki/Pseudorandom_number_generator https://en.wikipedia.org/wiki/Pseudorandom_number_generator
  - majormajor 2y ago
    That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation. Consider the errors like "this math library will have this specific function" (based on a hundred other math libraries for other languages usually having that).
    - AdieuToLogic 2y ago
      > That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation. I believe we are saying the same thing here. My clarification to the OP's statement: What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... Was that the algorithm has no concept of correctness (nor the other anthropomorphic attributes cited), but instead relies on pseudo-randomness to vary search paths when generating text.
grey-area 2y ago
LLMs are incapable of solving even simple logic puzzles or maths puzzles they haven't seen before, they don't have a model of the world which is key to intelligence. What they are good at is reproducing things in their dataset with slight modifications and (sometimes) responding to queries well which make them seem creative but there is no understanding or intelligence there, in spite of appearances. They are very good at fooling people; perhaps Turing's Test is not a good measure of intelligence after all, it can easily be gamed and we find it hard to differentiate apparent facility with language and intelligence/knowledge.
- jychang 2y ago
  LLMs clearly do have a world model though. They represent those ideas at higher level features in the feedforward layer. The lower level layers are neurons that describe words, syntax, and local structures in the text, while the upper levels capture more abstract ideas, such as semantic meaning, relationships between concepts, and even implicit reasoning patterns.
  - troupo 2y ago
    Funny how literally nothing of what you wrote is happening.
    - gmadsen 2y ago
      https://www.anthropic.com/news/mapping-mind-language-model https://www.anthropic.com/news/mapping-mind-language-model It is clearly happening as shown by numerous papers studying it. Here is a popular one by anthropic
      - troupo 2y ago
        > https://www.anthropic.com/news/mapping-mind-language-model https://www.anthropic.com/news/mapping-mind-language-model I wouldn't read into marketing materials by the people whose funding depends on hype. Nothing in the link you provided is even close to "neurons, model of the world, thinking" etc. It literally is "in our training data similar concepts were clustered with some other similar concepts, and manipulating these clusters lead to different outcomes".

admiralrohan 2y ago

Hallucinating is fine but overconfidence is the problem. But I heard it's not an easy problem to solve.

jacksnipe 2y ago
Unfortunately, in the system most of us work in today, I think overconfidence is an intelligent behavior.
- onemoresoop 2y ago
  I find it extremely dumb to see overconfident people that really have nothing special about them or are even incompetent. These people are not contributing positively to the system, quite on the contrary.
  - xwolfi 2y ago
    But we don't work for the system, fundamentally, we work for ourselves, and the system incentivizes us to work for it by aligning our constraints: if you work that direction, you'll get that reward. Overconfident people ofc do not contribute positively to the system, but they skew the system reward's calculation towards them: I swear I've done that work in that direction, where's my reward ? In a sense, they are extremely successful: they managed to do very low effort, get very high reward, help themselves like all of us but at a much better profit margin, by sacrificing a system that, let's be honest, none of us care about really. Your problem maybe, is that you swallowed the little BS the system fed you while incentivizing you: that the system matters more than yourself, at least at a greater extent than healthy ? And you see the same thing with AI: these things convince people so deeply of their intelligence that it blew to such proportion that NVidia is now worth trillions. I had a colleague mumbling yesterday that his wife now speaks more with ChatGPT than him. Overconfidence is a positive attribute... for oneself.
    - onemoresoop 2y ago
      Not talking about THE system or any system in particular same way gaming the system doesn’t refer to any system but just cheating in general. And if you like overconfident people good for you, I can’t stand them because they’re hollow with no basis in reality, flawed like everyone, just pumping out their egos with hot air. And your reasoning that overconfidence is a positive attribute doesn’t make much sense to me but we’re entitled to our own opinions.
    - butlike 2y ago
      Overconfident people are conquerors. Conquerors do not contribute positively to a harmonious system, true, but I'm not so sure we can glean the system is supposed to be harmonious. If one contributes "positively" to the system, everyone's value increases and the solution becomes more homogenized. Once the system is homogenized enough, it becomes vulnerable to adversity from an outside force. If the system is not harmonious/non-homoginized, the attacker would be drawn to the most powerful point in the system. Overconfident people aren't evil, they're simply stressing the system to make sure it can handle adversity from an outside force. They're saying: "listen, I'm going to take what you have, and you should be so happy that's all I'm taking." So I think overconfidence is a positive attribute for the system as well as for the overconfident individual. It's not a positive attribute for the local parties getting run over by the overconfident individual.
Terr_ 2y ago
> overconfidence is the problem. The problem is a bit deeper than that, because what we perceive as "confidence" is itself also an illusion. The (real) algorithm takes documents and makes them longer, and some humans configured a document that looks like a conversation between "User" and "AssistantBot", and they also wrote some code to act-out things that look like dialogue for one of the characters. The (real) trait of confidence involves next-token statistics. In contrast, the character named AssistantBot is "overconfident" in exactly the same sense that a character named Count Dracula is "immortal", "brooding", or "fearful" of garlic, crucifixes, and sunlight. Fictional traits we perceive on fictional characters from reading text. Yes, we can set up a script where the narrator periodically re-describes AssistantBot as careful and cautious, and that might help a bit with stopping humans from over-trusting the story they are being read. But trying to ensure logical conclusions arise from cautious reasoning is... well, indirect at best, much like trying to make it better at math by narrating "AssistantBot was good at math and diligent at checking the numbers." > Hallucinating P.S.: "Hallucinations" and prompt-injection are non-ironic examples of "it's not a bug, it's a feature". There's no minor magic incantation that'll permanently banish them without damaging how it all works.
- plaguuuuuu 2y ago
  I'd love to know if the conversational training set includes documents where the AI throws its hands up and goes "actually I have no idea". I'm guessing not.
  - Terr_ 2y ago
    There's also the problem of whether the LLM would learn to generate stories where the AssistantBot gives up in cases that match our own logical reasons, versus ones where the AssistantBot gives up because that's simply what AssistantBots in training-stories usually do when the User character uses words of disagreement and disapproval.
patates 2y ago
Hallucinating is a confidence problem, no? Say, they should be 100% confident that "0.3" follows "0.2 + 0.1 =", but a lot of floating point examples on the internet make them less confident. On a much more nuanced problem, "0.30000000000000004" may get more and more confidence. This is what makes them "hallucinate", did I get it wrong? (in other words, am I hallucinating myself? :) )

pydry 2y ago

I dunno hallucinations seem like a pretty human type of mistake to me. when i try to remember something my brain often synthesizes new things by filling in the gaps. This would be where I often say "i might be imagining it, but..." or "i could have sworn there was a..." In such cases the thing that saves the human brain is double checking against reality (e.g. googling it to make sure). Miscounting the number of r's in strawberry by glancing at the word also seems like a pretty human mistake.

gitaarik 2y ago
But it's different kinds of hallucinations. AI doesn't have a base understanding of how physics work. So they think it's acceptible if in a video some element on the background in a next frame might appear in front of another element that is on the foreground. So it's always necessary to keep correcting LLMs, because they only learn by example, and you can't express any possible outcome of any physical process just by example, because physical processes can be in infinate variations. LLMs can keep getting closer to match our physical reality, but when you zoom into the details you'll always find that it comes short. So you can never really trust an LLM. If we want to make an AI that doesn't make errors, it should understand how physics works.
- pydry 2y ago
  I dont think the errors really are all that different. Ever since GPT-3.5 came out Ive been thinking that the errors were ones a human could have made under a similar context. >LLMs can keep getting closer to match our physical reality, but when you zoom into the details you'll always find that it comes short. Like humans. >So you can never really trust an LLM. Cant really trust a human either. That's why we set up elaborate human systems (science, checks and balances in government, law, freedom of speech, markets) to mitigate our constant tendency to be complete fuck ups. We hallucinate science that does not exist, lies to maintain our worldview, jump to conclusions about guilt, build businesses based upon bad beliefs, etc. >If we want to make an AI that doesn't make errors, it should understand how physics works An AI that doesnt make errors wouldnt be AGI it would be a godlike superintelligence. I dont think thats even feasible. I think a propensity to make errors is intrinsic to how intelligence functions. Physics is just one domain that they work in and Im pretty sure some of them already do have varying understandings of physics.

j45 2y ago

Some of the errors are caused by humans. Say, due to changing the chat to only pay attention to recent messages and not the middle, omitting critical details.

ForTheKidz 2y ago

> I think of them as similar to human optical illusions. What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans than anything to do with what we refer to as "hallucinations" in humans—only they don't have the ability to analyze whether or not they're making up something or accurately parameterizing the vibes. The only connection between the two concepts is that the initial imagery from DeepDream was super trippy.

AdieuToLogic 2y ago
> What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... No. What people call LLM "hallucinations" is the result of a PRNG[0] influencing an algorithm to pursue a less statistically probable branch without regard nor understanding. 0 - https://en.wikipedia.org/wiki/Pseudorandom_number_generator https://en.wikipedia.org/wiki/Pseudorandom_number_generator
- majormajor 2y ago
  That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation. Consider the errors like "this math library will have this specific function" (based on a hundred other math libraries for other languages usually having that).
  - AdieuToLogic 2y ago
    > That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation. I believe we are saying the same thing here. My clarification to the OP's statement: What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ... Was that the algorithm has no concept of correctness (nor the other anthropomorphic attributes cited), but instead relies on pseudo-randomness to vary search paths when generating text.

grey-area 2y ago

LLMs are incapable of solving even simple logic puzzles or maths puzzles they haven't seen before, they don't have a model of the world which is key to intelligence. What they are good at is reproducing things in their dataset with slight modifications and (sometimes) responding to queries well which make them seem creative but there is no understanding or intelligence there, in spite of appearances. They are very good at fooling people; perhaps Turing's Test is not a good measure of intelligence after all, it can easily be gamed and we find it hard to differentiate apparent facility with language and intelligence/knowledge.

jychang 2y ago
LLMs clearly do have a world model though. They represent those ideas at higher level features in the feedforward layer. The lower level layers are neurons that describe words, syntax, and local structures in the text, while the upper levels capture more abstract ideas, such as semantic meaning, relationships between concepts, and even implicit reasoning patterns.
- troupo 2y ago
  Funny how literally nothing of what you wrote is happening.
  - gmadsen 2y ago
    https://www.anthropic.com/news/mapping-mind-language-model https://www.anthropic.com/news/mapping-mind-language-model It is clearly happening as shown by numerous papers studying it. Here is a popular one by anthropic
    - troupo 2y ago
      > https://www.anthropic.com/news/mapping-mind-language-model https://www.anthropic.com/news/mapping-mind-language-model I wouldn't read into marketing materials by the people whose funding depends on hype. Nothing in the link you provided is even close to "neurons, model of the world, thinking" etc. It literally is "in our training data similar concepts were clustered with some other similar concepts, and manipulating these clusters lead to different outcomes".