8 ms·

LLMs do not confabulate or hallucinate. They predict the next word in a sequence based on correlations derived from a lot of training data. Sometimes that data

by lsy 3y ago

LLMs do not confabulate or hallucinate. They predict the next word in a sequence based on correlations derived from a lot of training data. Sometimes that data happens to be structured enough to statistically generate a word that is fluent and corresponds to what we as readers understand as reality. Sometimes, due to the unclear nature of "truth" and the necessarily incomplete data used to train the model, the word or words that are statistically generated are merely fluent but do not correspond to reality. The answer to "What are LLMs but humans with extreme amnesia and no central coherence?" is that they are not like humans at all, and only resemble them superficially due to our anthropomorphic tendency to infer a mind upon seeing fluent and roughly correspondent text.

ta8645 3y ago
> they are not like humans at all, and only resemble them superficially due to our anthropomorphic tendency to infer a mind upon seeing fluent and roughly correspondent text. Alternatively, we overestimate human abilities, and imbue our thoughts with deep meaning and significance, simply as a way to feel more important than we actually are. Ultimately, we may rely on statistical prediction quite a bit as well. Obviously, LLMs aren't at the level of human cognition, but the impressive results they do produce, given their obvious simplicity, hints that intelligence may not be as hard to reproduce in silicon as we have previously assumed.
- Jensson 3y ago
  > Obviously, LLMs aren't at the level of human cognition, but the impressive results they do produce, given their obvious simplicity, hints that intelligence may not be as hard to reproduce in silicon as we have previously assumed. People said this about calculators as well, or initial chess AI as well. "If computers can do this, then the remaining parts of intelligence is probably not hard to reproduce!". So we already know many humans massively underestimate the depth of human intelligence. It happened many times before and will continue to happen every time we make computers do anything new.
  - ta8645 3y ago
    > People said this about calculators as well, or initial chess AI as well. "If computers can do this, then the remaining parts of intelligence is probably not hard to reproduce!". People also said that chess computers would never beat the best humans. And then when that happened they said, "well they're just using sheer brute force, they will never produce beautiful chess". And then Alpha Zero came along, and showed that prediction was wrong as well. Personally, I believe that silicon intelligence will evolve a lot faster than ours did. But regardless, I've yet to hear any convincing argument why it can't happen eventually.
    - Jensson 3y ago
      > I've yet to hear any convincing argument why it can't happen eventually. That is a strawman, basically nobody argues this. > People also said that chess computers would never beat the best humans How many did say this? I see way more overly optimistic "humans are stupid" messages than I see "computers will never beat humans" messages. The argument is mostly between "humans are obsolete in a decade at most" versus "we would be lucky to see general intelligence in our lifetime", not that it it is impossible.
      - ta8645 3y ago
        > That is a strawman, basically nobody argues this. Some people do. Some people believe that intelligence is much more fundamental to the universe, and not capable of being reproduced mechanistically. But if you grant that it is possible, all we're talking about is when; which is anyone's guess. It could be tomorrow, it could take a million years. I'm not pretending to know, just offering a different perspective than the original comment.
        Jensson 3y ago
        But those "some people" aren't here. What we are discussing is how close LLM are to human reasoning. Many argue LLM are already there, "what evidence do you have that humans aren't essentially just an LLM?" basically. You see such posts in this very thread.
- Guvante 3y ago
  Being able to mix what you have seen before isn't an accurate representation of any popular model for human intelligence to my knowledge. It is akin to a backronym "well that sounds similar so is the same" while ignoring the real differences.
  - ta8645 3y ago
    > Being able to mix what you have seen before But humans clearly do that. It's not all we do, but we clearly do that. And given the impressive results that LLMs produce, my guess is it or something like it, will play a significant role in the AGI of the future.
    - XorNot 3y ago
      I'd say possibly it is all we do, we just have a lot more hidden inputs. i.e. we're all dealing with inputs from our physical bodies now, and can't not be. There's a whole bit of social advice about dealing with people which is "remember you may be encountering someone on the worst day of their life" (the idea being, 95+% of the time you shouldn't escalate, particularly with strangers). What I do now, and how I think about things is being affected in ways I know are directly related to blood sugar, fatigue, pain (i.e. my legs are a little uncomfortable in a way yours aren't, but I moved them just now so they feel better - all unique inputs to me, all always there). But the other problem with the notion is people thinking of the knowledge bounds these systems derive as being "limits" - they're not. The whole point of machine learning is they're not interpolating between known datapoints, but rather extracting the rules which fit those data points. So stay between the points, and sure - it's some type of complex interpolation of seen inputs. But you don't have to stay between those datapoints - you can move of either end and extrapolate new ones entirely.
    - Guvante 3y ago
      I would say LLM is at best a precursor for a memory system. But saying we almost have AGI because we have a template for HDDs isn't exactly representative IMHO.
  - lern_too_spel 3y ago
- ffwd 3y ago
  To me predicting the next token is obviously not how humans think. If I ask you to envision a green triangle and a red square next to each other, and then swap the shapes but keep the colors in the same locations, and answer what color is the triangle now, you say the triangle is red, but you do so because you envisioned the triangle swapping places and did the mental steps etc. An LLM if even answering correctly, is statistically answering based on billions of lines of text + rlhf and all of this, I highly doubt there is a mental model of the world, but rather a large set of constraints in the probabilities which leads to the resulting answer. The reasoning ability is a secondary effect of the probabilities which is why it's hard to make it so every probability is correct for every answer I think. And regarding OP about hallucinating vs confabulating. To me hallucinating is a fine word for it, because it is filling in a gap or there aren't enough constraints in the model/data/tuning to account for that specific answer that it gave that was incorrect. Hence it "hallucinates" something in the gap. The real power of LLM's is that it seems to accumulate these 'constraints' (generalization), so that with the right model, it should be able to answer more and more prompts correctly, which is kind of amazing. Confabulation works too but is a little more high level IMO.
  - klik99 3y ago
    I'm not entirely sure if the mental model is somehow a layer deeper than the prediction that humans do. I used to believe it, and still use it as shorthand, but these days I'm not sure it's accurate. The triangle example doesn't prove it because our predictive model could also say "hey, things don't just change color and shape like that, they need to move". It's similar to how an LLM can be more accurate with math when asked to step through and reason through it's logic - by stepping through the individual steps it can create a larger system. The thing that made me question if a "mental model" is at the base of human cognition - people who do those memory competitions, the clear winning strategy is the memory palace, or imagining walking through a house where each room is another number - they have to build step by step memory, it's not like an SQL database where they can just SELECT from random. Another one was the insight from GTD that if you remember 7 things to do, you're always repeating those 7 things to yourself to keep them in active memory. There's a strong argument that the mental model is derivative of a predictive model in the human brain, and we can just appear to have a mental model since we have an internal dialogue that runs so fast in the background that even we rarely recognize it. (anyone who has kept a steady meditation or similar practice should be familiar with it)
krapp 3y ago
Yes, this is the problem. We have to be able to efficiently describe what LLMs do and how they do it, when what they do is superficially familiar, but how is fundamentally alien. We haven't yet developed the necessary language to discuss them on their own terms. Anthropomorphism and resorting to metaphors like "hallucinate" and "confabulate" are inevitable if you don't want to have to preface every comment with a paragraph of technical discussion. They get the necessary point across which is that the "reality" LLMs construct is not necessarily tethered to actual reality. They're deceptively convincing but can't be trusted.
- HybridCurve 3y ago
  I fully agree with this perspective. The terminology will change as the field continues to evolve. As long as any anthropomorphizing terms are chosen carefully and are not aggrandizing, it shouldn't be a problem. IIRC "hallucinate" was a term previously used to describe characteristics other network types such as RBMs and had just been carried over to LLMs.
legutierr 3y ago
> The answer to "What are LLMs but humans with extreme amnesia and no central coherence?" is that they are not like humans at all, and only resemble them superficially. Is there any chance at all that we humans are are also just predicting the next word in the sentence? That the "voice in your head" that so many people report having is itself some kind of LLM that is, to some extent, driving their reasoning?
- Jensson 3y ago
  > Is there any chance at all that we humans are are also just predicting the next word in the sentence No, humans live in a world and use their intelligence to manipulate the world. When we generate word sequence we usually generate those to communicate information about the world, not to try to parrot what others have said. Humans can use their intelligence to parrot what others say, many do, but we know for sure it is not all humans do.
  - kdmccormick 3y ago
    I fail to see how this makes us different from LLMs. LLMs are a bunch of computation running on silicon in order to predict the next word, and out of that all sorts of behavior arises, including intelligence, insigt,creativity, humor, and confabluation. Human brains are a bunch of computation running on carbon in order to <<X>>, and out of that all sorts of behavior arises, including intelligence, insigt,creativity, humor, and confabluation. We don't really know what <<X>> is. Why couldn't <<X>> be prediction as well?
  - legutierr 3y ago
    This seems like circular reasoning to me. "Human's don't just parrot what other people say, therefore we must be doing something more than what LLMs do, because an LLM only parrots the words that it is trained on. But an LLM is not truly intelligent because it is not doing what humans are doing—and because the LLM is not intelligent, all it is doing is parroting the training data." Where in this line of reasoning is it proved that the mechanisms are different? What if, instead, the conscious intelligence that differentiates humans from other animals is an emergent quality of language learning, specifically? Isn't there evidence that children that don't have access to language—deaf children who are not taught sign language, or feral children, or seriously neglected children—often cannot reason before they are taught language, and suffer from long-term cognitive impairment for having lacked language during their early development? Wouldn't this point to a fundamental linkage between language and human intelligence?
kromem 3y ago
"They predict the next word in a sequence based on correlations derived from a lot of training data." This misinformation really needs to finally die. They are trained based on correlations of next token prediction. But that does not mean that the resulting neural network is only doing that. When Harvard and MIT researchers fed Othello moves into GPT, it was training based on the correlations of legal Othello moves. But the version of the NN that performed the best at that task had created within that network dedicated structure representing a legal Othello board and tracking the state of moves - neither of which were things explicitly in the training data or goals the training was directly measuring. You effectively came to exist by a training process of surviving to reproduce. But it wouldn't exactly be accurate to say that as complexity increased, the only thing you remain capable of is surviving to reproduce, even if many of your emergent capabilities happen to improve the success of being able to do so. Training =/= Operation
- machiaweliczny 3y ago
  Yeah, i think that neural networks learn because they can represent if/else relation (via step function (activation)) and that makes them learn good heuristics at least. I wonder if someone can prove that NN can generate arbitrary programs because as we know 2 layer NNs can approxomate any calculation. Given some state and how easy is to create turing complete machine i think they already might have it. I speculate that the trick is that these programs sometimes dont evaluate in time when response is short. That’s why LLMs should be given way to adjust output and something like stop signal. Adjusting output can be done using DiffusER approach but not sure how to handle stop signal. I guess we could try to add fake input tokens for silience/thinking.
bigyikes 3y ago
>They predict the next word in a sequence ... Every time my mind takes this overplayed sequence of words as input, I can't help but feel the author refutes their own point...
BoorishBears 3y ago
Most of the people who repeat this have hardly any background in neuroscience... so they actually have no clue at all what the process they're claiming LLMs don't mirror is. Meanwhile the people who are in neuroscience have posited a Bayesian model to human cognitive function as far back as the 1800s, well before any convenient targets for anthropomorphization existed, and it continued as an area of study until today: https://global.oup.com/academic/product/bayesian-rationality-9780198524496 https://global.oup.com/academic/product/bayesian-rationality... Essentially, in typical techie fashion people with barely any understanding of a topic assume to have mastery ahead of people are the forefront of a non-tech field. _ This a sort of objectification seems equally as rooted in emotion and feelings as the anthropropmization crowd: We don't actually know the answer to the question of if there is some layer of human cognitive function that mirrors LLMs, but it'd cause me great cognitive dissonance/discomfort if the answer is not <insert preferred answer> so I will vehemently fight against any claims to the contrary. To me the answer is to simply accept we don't know, and take whatever tools let us make progress on working with the LLM. If that's borrowing words like "reasoning" and "understanding" and "confabulate" so be it, it's of no loss to anyone.
- krainboltgreene 3y ago
  You fundamentally do not understand the works you are linking to and it's hilarious that you would use it as a way of saying that neuroscientists think otherwise. I would note for other readers that the commenter didn't go to college for neuroscience, or any science, based on their previous comments.
  - BoorishBears 3y ago
    You didn't understand what I said, what the link implied, or how they were related, yet you forced a reply. I hope that your insecurity was patched over at least as long as it took to write the comment: that might partially offset the time anyone else takes to read it.
CapsAdmin 3y ago
I don't know exactly how humans think or how LLM's "predict the next token", but your argument is just refuting a high level description of behavior in place of a low level description. To me at least, it's a bit like saying a car's wheel do not spin when stepping on the pedal, it's the petrol engine that makes the wheel spin. You can replace petrol engine with an electric engine, and the description about the wheel spinning when stepping on the pedal is still technically correct regardless of any lower level explanation. We don't know how to describe the behavior of LLM's because of how foreign they are to us right now. Hallucinations are confabulations are meant to describe human behavior of couse, so the downside is it might make us anthropomorphize LLM's. However it's the best we could come up with I guess. If confabulation is a more precise explanation of behavior then I don't see why that's a bad idea.
- inopinatus 3y ago
  We do know how to describe their behaviour. It's in terms of plausibility. The central dogma of LLM-as-chatbot-assistant is that at sufficient scale, plausibility converges towards accuracy, and becomes a proxy for utility. This is not proven, but that is incidental to the deeper issue. The deep issue is that when a "conversation" kicks off with the LLM being confidently incorrect, the most plausible continuation to take from human literature, and from a good deal of human interaction, is that they continue to be wrong.
  - olalonde 3y ago
    I don't think it's that deep of an issue, or at least OpenAI seems to know how to fix it. ChatGPT often tell me when it's wrong and it also often recognizes that it has made a mistake when prompted (e.g. "are you sure?").
    - inopinatus 3y ago
      Yes, I too have seen ChatGPT issue an apology. Then it gives a worse version of its previous answer, buggier code etc. And it's because the most plausible continuation of that interaction is now of an ongoing conversation with an apologetic incompetent. Do not anthropomorphize the LLM. It does not have mental state. It does not have feelings. It does not have a suddenly realised goal of "doing better". The written apology was merely the most plausible text to issue at that point. It will then go on to simulate being wrong again, which I've come to recognise is more plausible than an idiot suddenly becoming a wizard. In such circumstances, the actual best remedy is to start afresh with a revised prompt.
p-e-w 3y ago
> they are not like humans at all, and only resemble them superficially We have no idea how LLMs work on a high level, and we have no idea how the human mind works on a high level. Therefore, such claims are rather overconfident. The fact that the two are dissimilar at the plumbing layer doesn't mean they cannot be alike in how they "really" operate. Either way, we simply do not know, so any such talk is (bad) philosophy, not science.
- krainboltgreene 3y ago
  > We have no idea how LLMs work on a high level This is just incredibly wrong. Do you think that LLM's were created out of thin air? A random combination of bytes that we happened to discover that we also happen to be improving/changing?
  - frabcus 3y ago
    We understand the training algorithm, but not the network that was trained. There’s the whole interpretability sub-discipline which has limited but interesting results. Those just indicate even more that we really have no idea how something like GPT-4 works inside its trained model.
  - p-e-w 3y ago
    Nobody knows how LLMs work on a high level. There's an algorithm that consumes Terabytes of raw text and produces 175 billion parameters, that when used as a statistical model magically reproduce human language. But the conceptual connection between those parameters and what the model can do is a complete mystery.
SubiculumCode 3y ago
Confabulation in humans seems much like word/sequence completion, autocompleting from gist information without reality monitoring, executive control, etc. Likewise, introducing output monitoring that can identify signals of recall failure, and executive control (stop signals), probably could help LLMs too. One of my research chapters covered some of these ideas, but the wikipedia does pretty well too. https://en.m.wikipedia.org/wiki/Confabulation https://en.m.wikipedia.org/wiki/Confabulation
Falkon1313 3y ago
That's exactly it, and that is what is so hard to get other people to understand. It's a random word generator. It can't 'hallucinate' or 'confabulate' anymore than dice can. It cannot possibly be wrong or incorrect. Those of us who've played D&D or something similar are familiar with rolling dice, say a 10-sided die and a 6-sided die, and cross-referencing the results against a table to generate random monster encounters, treasure, etc. If you roll a die to determine how many gold pieces an orc is carrying, and you roll a 5, that is the result. You don't claim that the dice was hallucinating or confabulating or got the facts wrong or whatever. It's a random generator, and it generated a random number, and that result is truthfully the result it generated. LLMs are also random generators. They're not databases of facts, they're not mathematical calculators. They are like the D&D random generation tables, only developed deeply enough such that they can string together randomly generated words into coherent sentences and paragraphs. That sentence/paragraph is just as true as the result of rolling some dice. If you roll an 11, then you rolled an 11. The dice didn't hallucinate it, confabulate it, lie about it, or get it wrong. The true result is that you rolled an 11. If you wanted to roll a 12, too bad. You were unlucky. The dice weren't somehow magically 'wrong'. And an LLM is not somehow magically wrong or incorrect or hallucinating or even confabulating when it gives you an answer that isn't what you wanted. It's randomly generating text. And its answer is every bit as correct as the result of rolling dice or drawing a card.
- travisjungroth 3y ago
  Have you used an LLM before? > They are like the D&D random generation tables, only developed deeply enough such that they can string together randomly generated words into coherent sentences and paragraphs. https://chat.openai.com/share/cd4c8c88-f702-4eaf-9f5f-d212f2cc1b47 https://chat.openai.com/share/cd4c8c88-f702-4eaf-9f5f-d212f2... Here’s an example where ChatGPT wrote code that solves the problem and then makes a bunch of statements that are all true. I can’t imagine any of those statements were even in the training data. I don’t see how you can say this is the same as drawing a card out of a deck. You should stop trying to get people to understand that. It’s not true.
nologic01 3y ago
This is a beautiful summary of what happens in reality with these statistical algorithms applied to digitized representations of human language. But as you can see in the comments, people simply do not want to internalize the mundane truth. Confabulating fake AGI narratives and hallucinating AI supremacy is more fun (also more profitable).
KaiserPro 3y ago
Exactly. The relentless "humanising" of LLMs really doesn't help. Its a noisy signal generated from either imprecise data, or not enough context to make a valid token choice.

by lsy 3y ago

ta8645 3y ago

> they are not like humans at all, and only resemble them superficially due to our anthropomorphic tendency to infer a mind upon seeing fluent and roughly correspondent text. Alternatively, we overestimate human abilities, and imbue our thoughts with deep meaning and significance, simply as a way to feel more important than we actually are. Ultimately, we may rely on statistical prediction quite a bit as well. Obviously, LLMs aren't at the level of human cognition, but the impressive results they do produce, given their obvious simplicity, hints that intelligence may not be as hard to reproduce in silicon as we have previously assumed.

Jensson 3y ago
> Obviously, LLMs aren't at the level of human cognition, but the impressive results they do produce, given their obvious simplicity, hints that intelligence may not be as hard to reproduce in silicon as we have previously assumed. People said this about calculators as well, or initial chess AI as well. "If computers can do this, then the remaining parts of intelligence is probably not hard to reproduce!". So we already know many humans massively underestimate the depth of human intelligence. It happened many times before and will continue to happen every time we make computers do anything new.
- ta8645 3y ago
  > People said this about calculators as well, or initial chess AI as well. "If computers can do this, then the remaining parts of intelligence is probably not hard to reproduce!". People also said that chess computers would never beat the best humans. And then when that happened they said, "well they're just using sheer brute force, they will never produce beautiful chess". And then Alpha Zero came along, and showed that prediction was wrong as well. Personally, I believe that silicon intelligence will evolve a lot faster than ours did. But regardless, I've yet to hear any convincing argument why it can't happen eventually.
  - Jensson 3y ago
    > I've yet to hear any convincing argument why it can't happen eventually. That is a strawman, basically nobody argues this. > People also said that chess computers would never beat the best humans How many did say this? I see way more overly optimistic "humans are stupid" messages than I see "computers will never beat humans" messages. The argument is mostly between "humans are obsolete in a decade at most" versus "we would be lucky to see general intelligence in our lifetime", not that it it is impossible.
    - ta8645 3y ago
      > That is a strawman, basically nobody argues this. Some people do. Some people believe that intelligence is much more fundamental to the universe, and not capable of being reproduced mechanistically. But if you grant that it is possible, all we're talking about is when; which is anyone's guess. It could be tomorrow, it could take a million years. I'm not pretending to know, just offering a different perspective than the original comment.
      - Jensson 3y ago
        But those "some people" aren't here. What we are discussing is how close LLM are to human reasoning. Many argue LLM are already there, "what evidence do you have that humans aren't essentially just an LLM?" basically. You see such posts in this very thread.
Guvante 3y ago
Being able to mix what you have seen before isn't an accurate representation of any popular model for human intelligence to my knowledge. It is akin to a backronym "well that sounds similar so is the same" while ignoring the real differences.
- ta8645 3y ago
  > Being able to mix what you have seen before But humans clearly do that. It's not all we do, but we clearly do that. And given the impressive results that LLMs produce, my guess is it or something like it, will play a significant role in the AGI of the future.
  - XorNot 3y ago
    I'd say possibly it is all we do, we just have a lot more hidden inputs. i.e. we're all dealing with inputs from our physical bodies now, and can't not be. There's a whole bit of social advice about dealing with people which is "remember you may be encountering someone on the worst day of their life" (the idea being, 95+% of the time you shouldn't escalate, particularly with strangers). What I do now, and how I think about things is being affected in ways I know are directly related to blood sugar, fatigue, pain (i.e. my legs are a little uncomfortable in a way yours aren't, but I moved them just now so they feel better - all unique inputs to me, all always there). But the other problem with the notion is people thinking of the knowledge bounds these systems derive as being "limits" - they're not. The whole point of machine learning is they're not interpolating between known datapoints, but rather extracting the rules which fit those data points. So stay between the points, and sure - it's some type of complex interpolation of seen inputs. But you don't have to stay between those datapoints - you can move of either end and extrapolate new ones entirely.
  - Guvante 3y ago
    I would say LLM is at best a precursor for a memory system. But saying we almost have AGI because we have a template for HDDs isn't exactly representative IMHO.
- lern_too_spel 3y ago
ffwd 3y ago
To me predicting the next token is obviously not how humans think. If I ask you to envision a green triangle and a red square next to each other, and then swap the shapes but keep the colors in the same locations, and answer what color is the triangle now, you say the triangle is red, but you do so because you envisioned the triangle swapping places and did the mental steps etc. An LLM if even answering correctly, is statistically answering based on billions of lines of text + rlhf and all of this, I highly doubt there is a mental model of the world, but rather a large set of constraints in the probabilities which leads to the resulting answer. The reasoning ability is a secondary effect of the probabilities which is why it's hard to make it so every probability is correct for every answer I think. And regarding OP about hallucinating vs confabulating. To me hallucinating is a fine word for it, because it is filling in a gap or there aren't enough constraints in the model/data/tuning to account for that specific answer that it gave that was incorrect. Hence it "hallucinates" something in the gap. The real power of LLM's is that it seems to accumulate these 'constraints' (generalization), so that with the right model, it should be able to answer more and more prompts correctly, which is kind of amazing. Confabulation works too but is a little more high level IMO.
- klik99 3y ago
  I'm not entirely sure if the mental model is somehow a layer deeper than the prediction that humans do. I used to believe it, and still use it as shorthand, but these days I'm not sure it's accurate. The triangle example doesn't prove it because our predictive model could also say "hey, things don't just change color and shape like that, they need to move". It's similar to how an LLM can be more accurate with math when asked to step through and reason through it's logic - by stepping through the individual steps it can create a larger system. The thing that made me question if a "mental model" is at the base of human cognition - people who do those memory competitions, the clear winning strategy is the memory palace, or imagining walking through a house where each room is another number - they have to build step by step memory, it's not like an SQL database where they can just SELECT from random. Another one was the insight from GTD that if you remember 7 things to do, you're always repeating those 7 things to yourself to keep them in active memory. There's a strong argument that the mental model is derivative of a predictive model in the human brain, and we can just appear to have a mental model since we have an internal dialogue that runs so fast in the background that even we rarely recognize it. (anyone who has kept a steady meditation or similar practice should be familiar with it)

krapp 3y ago

Yes, this is the problem. We have to be able to efficiently describe what LLMs do and how they do it, when what they do is superficially familiar, but how is fundamentally alien. We haven't yet developed the necessary language to discuss them on their own terms. Anthropomorphism and resorting to metaphors like "hallucinate" and "confabulate" are inevitable if you don't want to have to preface every comment with a paragraph of technical discussion. They get the necessary point across which is that the "reality" LLMs construct is not necessarily tethered to actual reality. They're deceptively convincing but can't be trusted.

HybridCurve 3y ago
I fully agree with this perspective. The terminology will change as the field continues to evolve. As long as any anthropomorphizing terms are chosen carefully and are not aggrandizing, it shouldn't be a problem. IIRC "hallucinate" was a term previously used to describe characteristics other network types such as RBMs and had just been carried over to LLMs.

legutierr 3y ago

> The answer to "What are LLMs but humans with extreme amnesia and no central coherence?" is that they are not like humans at all, and only resemble them superficially. Is there any chance at all that we humans are are also just predicting the next word in the sentence? That the "voice in your head" that so many people report having is itself some kind of LLM that is, to some extent, driving their reasoning?

Jensson 3y ago
> Is there any chance at all that we humans are are also just predicting the next word in the sentence No, humans live in a world and use their intelligence to manipulate the world. When we generate word sequence we usually generate those to communicate information about the world, not to try to parrot what others have said. Humans can use their intelligence to parrot what others say, many do, but we know for sure it is not all humans do.
- kdmccormick 3y ago
  I fail to see how this makes us different from LLMs. LLMs are a bunch of computation running on silicon in order to predict the next word, and out of that all sorts of behavior arises, including intelligence, insigt,creativity, humor, and confabluation. Human brains are a bunch of computation running on carbon in order to <<X>>, and out of that all sorts of behavior arises, including intelligence, insigt,creativity, humor, and confabluation. We don't really know what <<X>> is. Why couldn't <<X>> be prediction as well?
- legutierr 3y ago
  This seems like circular reasoning to me. "Human's don't just parrot what other people say, therefore we must be doing something more than what LLMs do, because an LLM only parrots the words that it is trained on. But an LLM is not truly intelligent because it is not doing what humans are doing—and because the LLM is not intelligent, all it is doing is parroting the training data." Where in this line of reasoning is it proved that the mechanisms are different? What if, instead, the conscious intelligence that differentiates humans from other animals is an emergent quality of language learning, specifically? Isn't there evidence that children that don't have access to language—deaf children who are not taught sign language, or feral children, or seriously neglected children—often cannot reason before they are taught language, and suffer from long-term cognitive impairment for having lacked language during their early development? Wouldn't this point to a fundamental linkage between language and human intelligence?

kromem 3y ago

"They predict the next word in a sequence based on correlations derived from a lot of training data." This misinformation really needs to finally die. They are trained based on correlations of next token prediction. But that does not mean that the resulting neural network is only doing that. When Harvard and MIT researchers fed Othello moves into GPT, it was training based on the correlations of legal Othello moves. But the version of the NN that performed the best at that task had created within that network dedicated structure representing a legal Othello board and tracking the state of moves - neither of which were things explicitly in the training data or goals the training was directly measuring. You effectively came to exist by a training process of surviving to reproduce. But it wouldn't exactly be accurate to say that as complexity increased, the only thing you remain capable of is surviving to reproduce, even if many of your emergent capabilities happen to improve the success of being able to do so. Training =/= Operation

machiaweliczny 3y ago
Yeah, i think that neural networks learn because they can represent if/else relation (via step function (activation)) and that makes them learn good heuristics at least. I wonder if someone can prove that NN can generate arbitrary programs because as we know 2 layer NNs can approxomate any calculation. Given some state and how easy is to create turing complete machine i think they already might have it. I speculate that the trick is that these programs sometimes dont evaluate in time when response is short. That’s why LLMs should be given way to adjust output and something like stop signal. Adjusting output can be done using DiffusER approach but not sure how to handle stop signal. I guess we could try to add fake input tokens for silience/thinking.

bigyikes 3y ago

>They predict the next word in a sequence ... Every time my mind takes this overplayed sequence of words as input, I can't help but feel the author refutes their own point...

BoorishBears 3y ago

Most of the people who repeat this have hardly any background in neuroscience... so they actually have no clue at all what the process they're claiming LLMs don't mirror is. Meanwhile the people who are in neuroscience have posited a Bayesian model to human cognitive function as far back as the 1800s, well before any convenient targets for anthropomorphization existed, and it continued as an area of study until today: https://global.oup.com/academic/product/bayesian-rationality-9780198524496 https://global.oup.com/academic/product/bayesian-rationality... Essentially, in typical techie fashion people with barely any understanding of a topic assume to have mastery ahead of people are the forefront of a non-tech field. _ This a sort of objectification seems equally as rooted in emotion and feelings as the anthropropmization crowd: We don't actually know the answer to the question of if there is some layer of human cognitive function that mirrors LLMs, but it'd cause me great cognitive dissonance/discomfort if the answer is not <insert preferred answer> so I will vehemently fight against any claims to the contrary. To me the answer is to simply accept we don't know, and take whatever tools let us make progress on working with the LLM. If that's borrowing words like "reasoning" and "understanding" and "confabulate" so be it, it's of no loss to anyone.

krainboltgreene 3y ago
You fundamentally do not understand the works you are linking to and it's hilarious that you would use it as a way of saying that neuroscientists think otherwise. I would note for other readers that the commenter didn't go to college for neuroscience, or any science, based on their previous comments.
- BoorishBears 3y ago
  You didn't understand what I said, what the link implied, or how they were related, yet you forced a reply. I hope that your insecurity was patched over at least as long as it took to write the comment: that might partially offset the time anyone else takes to read it.

CapsAdmin 3y ago

I don't know exactly how humans think or how LLM's "predict the next token", but your argument is just refuting a high level description of behavior in place of a low level description. To me at least, it's a bit like saying a car's wheel do not spin when stepping on the pedal, it's the petrol engine that makes the wheel spin. You can replace petrol engine with an electric engine, and the description about the wheel spinning when stepping on the pedal is still technically correct regardless of any lower level explanation. We don't know how to describe the behavior of LLM's because of how foreign they are to us right now. Hallucinations are confabulations are meant to describe human behavior of couse, so the downside is it might make us anthropomorphize LLM's. However it's the best we could come up with I guess. If confabulation is a more precise explanation of behavior then I don't see why that's a bad idea.

inopinatus 3y ago
We do know how to describe their behaviour. It's in terms of plausibility. The central dogma of LLM-as-chatbot-assistant is that at sufficient scale, plausibility converges towards accuracy, and becomes a proxy for utility. This is not proven, but that is incidental to the deeper issue. The deep issue is that when a "conversation" kicks off with the LLM being confidently incorrect, the most plausible continuation to take from human literature, and from a good deal of human interaction, is that they continue to be wrong.
- olalonde 3y ago
  I don't think it's that deep of an issue, or at least OpenAI seems to know how to fix it. ChatGPT often tell me when it's wrong and it also often recognizes that it has made a mistake when prompted (e.g. "are you sure?").
  - inopinatus 3y ago
    Yes, I too have seen ChatGPT issue an apology. Then it gives a worse version of its previous answer, buggier code etc. And it's because the most plausible continuation of that interaction is now of an ongoing conversation with an apologetic incompetent. Do not anthropomorphize the LLM. It does not have mental state. It does not have feelings. It does not have a suddenly realised goal of "doing better". The written apology was merely the most plausible text to issue at that point. It will then go on to simulate being wrong again, which I've come to recognise is more plausible than an idiot suddenly becoming a wizard. In such circumstances, the actual best remedy is to start afresh with a revised prompt.

p-e-w 3y ago

> they are not like humans at all, and only resemble them superficially We have no idea how LLMs work on a high level, and we have no idea how the human mind works on a high level. Therefore, such claims are rather overconfident. The fact that the two are dissimilar at the plumbing layer doesn't mean they cannot be alike in how they "really" operate. Either way, we simply do not know, so any such talk is (bad) philosophy, not science.

krainboltgreene 3y ago
> We have no idea how LLMs work on a high level This is just incredibly wrong. Do you think that LLM's were created out of thin air? A random combination of bytes that we happened to discover that we also happen to be improving/changing?
- frabcus 3y ago
  We understand the training algorithm, but not the network that was trained. There’s the whole interpretability sub-discipline which has limited but interesting results. Those just indicate even more that we really have no idea how something like GPT-4 works inside its trained model.
- p-e-w 3y ago
  Nobody knows how LLMs work on a high level. There's an algorithm that consumes Terabytes of raw text and produces 175 billion parameters, that when used as a statistical model magically reproduce human language. But the conceptual connection between those parameters and what the model can do is a complete mystery.

SubiculumCode 3y ago

Confabulation in humans seems much like word/sequence completion, autocompleting from gist information without reality monitoring, executive control, etc. Likewise, introducing output monitoring that can identify signals of recall failure, and executive control (stop signals), probably could help LLMs too. One of my research chapters covered some of these ideas, but the wikipedia does pretty well too. https://en.m.wikipedia.org/wiki/Confabulation https://en.m.wikipedia.org/wiki/Confabulation

Falkon1313 3y ago

That's exactly it, and that is what is so hard to get other people to understand. It's a random word generator. It can't 'hallucinate' or 'confabulate' anymore than dice can. It cannot possibly be wrong or incorrect. Those of us who've played D&D or something similar are familiar with rolling dice, say a 10-sided die and a 6-sided die, and cross-referencing the results against a table to generate random monster encounters, treasure, etc. If you roll a die to determine how many gold pieces an orc is carrying, and you roll a 5, that is the result. You don't claim that the dice was hallucinating or confabulating or got the facts wrong or whatever. It's a random generator, and it generated a random number, and that result is truthfully the result it generated. LLMs are also random generators. They're not databases of facts, they're not mathematical calculators. They are like the D&D random generation tables, only developed deeply enough such that they can string together randomly generated words into coherent sentences and paragraphs. That sentence/paragraph is just as true as the result of rolling some dice. If you roll an 11, then you rolled an 11. The dice didn't hallucinate it, confabulate it, lie about it, or get it wrong. The true result is that you rolled an 11. If you wanted to roll a 12, too bad. You were unlucky. The dice weren't somehow magically 'wrong'. And an LLM is not somehow magically wrong or incorrect or hallucinating or even confabulating when it gives you an answer that isn't what you wanted. It's randomly generating text. And its answer is every bit as correct as the result of rolling dice or drawing a card.

travisjungroth 3y ago
Have you used an LLM before? > They are like the D&D random generation tables, only developed deeply enough such that they can string together randomly generated words into coherent sentences and paragraphs. https://chat.openai.com/share/cd4c8c88-f702-4eaf-9f5f-d212f2cc1b47 https://chat.openai.com/share/cd4c8c88-f702-4eaf-9f5f-d212f2... Here’s an example where ChatGPT wrote code that solves the problem and then makes a bunch of statements that are all true. I can’t imagine any of those statements were even in the training data. I don’t see how you can say this is the same as drawing a card out of a deck. You should stop trying to get people to understand that. It’s not true.

nologic01 3y ago

This is a beautiful summary of what happens in reality with these statistical algorithms applied to digitized representations of human language. But as you can see in the comments, people simply do not want to internalize the mundane truth. Confabulating fake AGI narratives and hallucinating AI supremacy is more fun (also more profitable).

KaiserPro 3y ago

Exactly. The relentless "humanising" of LLMs really doesn't help. Its a noisy signal generated from either imprecise data, or not enough context to make a valid token choice.