7 ms·

For anyone who still doesn't understand why more and more folks are pointing out that LLMs "hallucinate" 100% of the time, let me put it to you this way: what i

• throw46365 2y ago
  > If there is a difference, where does that exist? In the mechanism of the LLM, or in your mind? Thank you for this sentence: it is hard to get across how often Gen-AI proponents are actually projecting perceived success onto LLMs while downplaying error.
  • empath75 2y ago
    I mostly see the reverse.
    • throw46365 2y ago
      You mostly see people projecting perceived error onto LLMs? I don't think I've seen a single article about an AI getting things wrong, recently, where there was a nuanced notion about whether it was actually wrong. I don't think we're anywhere close to "nuanced mistakes are the main problem" yet.
      • empath75 2y ago
        I mostly see people ignoring successes and playing up every error.
        • throw46365 2y ago
          But the errors are fundamental, and the successes actually subjective as a result. That is, it appears to get things right, really a lot, but the conclusions people draw about why it gets things right are undermined by the nature of the errors. Like, it must have a world model, it must understand the meaning of... etc.; the nature of the errors they are downplaying fundamentally undermines the certainty of these projections.
• sigmoid10 2y ago
  >what is the LLM doing differently when it generates tokens that are "wrong" compared to when the tokens are "right"? When an LLM is trained, it essentially compresses the knowledge of the training data corpus into a world model. "Right" and "false" are thereby only emergent when you have a different world model for yourself that tells you a different answer, most likely because the LLM was undertrained on the target domain. But to the LLM, the token with the highest probability will be the most likely correct answer, similarly to how you might have a "gut feeling" when asked about something which you clearly don't understand and have no education in. And you will be wrong just as often. The perceived overconfidence of wrong answers likely stems from human behaviour in the training data as well. LLMs are not better than humans, but they are also not worse. They are just a distilled encoding of human behaviour, which in turn might be all that the human mind is in the end.
  • smogcutter 2y ago
    > But to the LLM, the token with the highest probability will be the most likely correct answer This is precisely what people are identifying as problematic
  • swatcoder 2y ago
    No. LLM's become fluent in constructing coherent, sophisticated text in natural language from training on obscene amounts of coherent, sophisticated text in natural language. Importantly, there is no such corpus of text that contains only accurate knowledge, let alone knowledge as it unambiguously applies to some specific domain. It's unclear that any such corpus could exist (a millennias old discussion in philosophy with no possible resolution), but even if you take for granted that such a corpus could, we don't have one. So what happens is that after learning how to construct coherent, sophisticated text in natural language from all the bullshit-adled general text that includes truth and fiction and lies and fantasy and bullshit and garbage and old text and new text, there is a subsequent effort to sort of tune the model in on some generating useful text towards some purpose. And here, again, it's important to distinguish that this subsequent training is about utility ("you're a helpful chatbot", "this will trigger a function call that will supplement results", etc) and so still can't focus strictly on knowledge. LLM's can produce intelligent output that may be correct and may be verifiable, but the way they work and the way they need to be trained prevents them from ever actually representing knowledge itself. The best they can do is create text that is more or less fluent and more or less useful. It's awesome and has lots and lots of potential, but it's a radically different thing than a material individual that's composed of countless disparate linguistic and non-linguistic systems that have never yet been technologically replicated or modeled.
• temporarely 2y ago
  The terminology is wrong but your point is valid. There is no internal criteria or mechanism for statement verification. As the mind likely is also in part a high dimensional construct and LLMs to an extent represent our collective jumble of 'notions' it is natural that their emits resonate with human users. Q1: A ""correct" symbolic representation" of x. What is x? Your "Is there an intent to communicate, or" choice construct is problematic. Why would one require a "symbolic representation" of x, x likely being a 'meaningful thought'. So this is a hot debate whether semantics is primary or merely application. I believe it is primary in which case "symbolic representation" is 'an aid' to gaining a concrete sense of what is 'somehow' 'understood'. You observe a phenomena, and understand its dynamics. You may even anticipate it while observing. To formalize that understanding is the beginning of 'expression'. Q2: because while there is a function LLM(encodings, q) that emits 'plausible' responses, an equivalent function for Pi does not exist outside of 'pure inexpressible realm of understanding' :)
  • salawat 2y ago
    >I believe it is primary in which case "symbolic representation" is 'an aid' to gaining a concrete sense of what is 'somehow' 'understood'. There is nothing magic about perception to distinguish it meaningfully from symbolic representation; in point of fact, that which you experience is in and of itself a symbolic representation of the world around you. You do not sense the frequencies outside the physical transduction capabilities of the human ear, or the wavelengths similarly beyond the capability to transduce of the human eye, or feel distinct contact beyond the density of haptic transduction of somatic nerves. Nevertheless, those phenomena are still there, and despite their insensible nature, have an effect on you. Your entire perception is a map, which one would be well advised to not mistake for the territory. To dismiss symbolic representation as something that only occurs on communication after perception is to "lose sight" of the fact that all the data your mind integrates into a perception is itself, symbolic. Communication, and symbolic representation is all there is, and it happens long before we even get to the partnof reality where I'm trying to select words to converse about it with you.
    • Gormo 2y ago
• Drakim 2y ago
  What is the difference between a google map style application that shows pixels that are "right" for a road, and pixels that are "wrong" for a road? A pixel is a pixel, colors cannot be true or false. The only way we can say a pixel is right or wrong in google maps is whether the act of the human utilizing that pixel to understand geographical information results in the human correctly navigating the road. In the same way, an LLM can tell me all kinds of things, and those things are just words, which are just characters, which are just bytes, which are just electrons. There is no truth or false value to an electron flowing in a specific pattern in my CPU. The real question is what I, the human, get out of reading those words, if I end up correctly navigating and understanding something about the world based on what I read.
  • deleted 2y ago
    [deleted]
  • _boffin_ 2y ago
    But then again, that road, according to a neighboring country who thinks that it’s their land and it isn’t a road. Depending on what country you’re in, it makes a difference
  • tines 2y ago
    > The real question is what I, the human, get out of reading those words So then you agree with the OP that an LLM is not intelligent in the same way that Google Maps is not intelligent? That seems to be where your argument leads but you're replying in a way that makes me think you are disagreeing with the op.
    • contravariant 2y ago
      Any reply will always sound like someone is disagreeing, even if they claim not to. Though in this case I'm not even sure what the comment they're supposedly disagreeing with is even claiming. Is it even claiming anything?
      • tines 2y ago
• dangerwill 2y ago
  Do we actually think this way though? When I am talking with someone I am cognating about what information and emotion I want to impart to the person / thinking about how they are perceiving me and the sentence construction flows from these intents. Only the sentence construction is even analogous to token generation, and even then, we edit our sentences in our heads all the time before or while talking. Instead of just a constant forward stream of tokens from the LLM.
• jellicle 2y ago
  If I make a machine that makes random sounds in approximately the human vocal range, and occasionally humans who listen to it hear "words" (in their language, naturally), then is that machine telling the truth when words are heard and "hallucinating" the rest of the time?
  • bombcar 2y ago
    Now I'm imagining a device that takes as input the roaring of a furnace, and only outputs when it recognizes words.
• mcguire 2y ago
  (An aside: Writing is a representation of speech, not the other way around.)
• visarga 2y ago
  > what is the LLM doing differently when it generates tokens that are "wrong" compared to when the tokens are "right"? When they don't recall correctly, it is hallucination. When they recall perfectly, it is regurgitation/copyright infringement. We find issue either way. May I remind you that we also hallucinate, memory plays tricks on us. We often google stuff just to be sure. It is not the hallucination part that is a real difference between humans and LLMs. > Why do humans produce speech? We produce language to solve body/social/environment related problems. LLMs don't have bodies but they do have environments, such as a chat room, where the user is the environment for the model. In fact chat rooms produce trillions of tokens per month worth of interaction and immediate feedback. If you look at what happens with those trillions of tokens - they go into the heads of hundreds of millions of people, who use the LLM assistance to solve their problems, and of course produce real world effects. Then it will reflect in the next training set, creating a second feedback loop between LLM and environment. By the way, humans don't produce speech individually, if left alone, without humanity as support. We only learn speech when we get together. Language is social. Human brain is not so smart on its own, but language collects experience across generations. We rely on language for intelligence to a larger degree than we like to admit. Isn't it a mystery how LLMs learned so many language skills purely from imitating us without their own experience? It shows just how powerful language is on its own. And it shows it can be independent on substrate.
  • jgalt212 2y ago
    > When they don't recall correctly, it is hallucination. When they recall perfectly, it is regurgitation/copyright infringement. We find issue either way. You nailed it right there.
• gwern 2y ago
  > let me put it to you this way: what is the LLM doing differently when it generates tokens that are "wrong" compared to when the tokens are "right"? It is conditioning on latents about truth, falsity, reliability, and calibration. All of these inferred latents have been shown to exist inside LLMs, as they need to exist for LLMs to do their jobs in accurately predicting the next token. (Imagine trying to predict tokens in, say, discussions about people arguing or critiques of fictional stories, or discussing mistakes made by people, and not having latents like that!) LLMs also model other things: for example, you can use them to predict demographic information about the authors of texts (https://arxiv.org/abs/2310.07298 https://arxiv.org/abs/2310.07298), even though this is something that pretty much never exists IRL, a piece of text with a demographic label like "written by a 28yo"; it is simply a latent that the LLM has learned for its usefulness, and can be tapped into. This is why a LLM can generate text that it thinks was written by a Valley girl in the 1980s, or text which is 'wrong', or text which is 'right', and this is why you see things like in Codex, they found that if the prompted code had subtle bugs, the completions tend to have subtle bugs - because the model knows there's 'good' and 'bad' code, and 'bad' code will be followed by more bad code, and so on. This should all come as no surprise - what else did you think would happen? - but pointing out that for it to be possible, the LLM has to be inferring hidden properties of the text like the nature of its author, seems to surprise people.
  • timr 2y ago
    > It is conditioning on latents about truth, falsity, reliability, and calibration. All of these inferred latents have been shown to exist inside LLMs, as they need to exist for LLMs to do their jobs in accurately predicting the next token. No, it isn't, and no, they haven't [1], and no, they don't. The only thing that "needs to exist" for an LLM to generate the next token is a whole bunch of training data containing that token, so that it can condition based on context. You can stare at your navel and claim that these higher-level concepts end up encoded in the bajillions of free parameters of the model -- and hey, maybe they do -- but that's not the same thing as "conditioning on latents". There's no explicit representation of "truth" in an LLM, just like there's no explicit representation of a dog in Stable Diffusion. Do the thought exercise: if you trained an LLM on nothing but nonsense text, would it produce "truth"? LLMs "hallucinate" precisely because they have no idea what truth means. It's just a weird emergent outcome that when you train them on the entire internet, they generate something close to enough to truthy, most of the time. But it's all tokens to the model. [1] I have no idea how you could make the claim that something like a latent conceptualization of truth is "proven" to exist, given that proving any non-trivial statement true or false is basically impossible. How would you even evaluate this capability?
• Lerc 2y ago
  >what is the LLM doing differently when it generates tokens that are "wrong" compared to when the tokens are "right"? If there is a difference, where does that exist? In the mechanism of the LLM, or in your mind? If there were a detectable difference within the mechanism, the problem of hallucinations would be easy to fix. There may be ways to analyze logits to find patterns of uncertainty characteristics related to hallucinations. Perhaps deeper introspection of weights might turn up patterns. The difference isn't really in your mind either. The difference is simply that the one answer correlates with reality and the other does not. The point of AI models is to generalize from the training data, that implicitly means generating output that it hasn't seen as input. Perhaps the issue is not so much that it is generalizing/guessing but the degree to which making a guess is the right call is dependent on context.
• yarg 2y ago
  Bonus question 2 is the most ridiculous straw man I've seen in a very long time. The existence of arbitrary string encodings in transcendental numbers proves absolutely nothing about the processing capabilities of adaptive algorithms.
  • pennomi 2y ago
    Exactly. Reading digits of pi doesn’t converge toward anything. (And neither do infinite typewriter monkeys.) Any correct value they get is random, and exceedingly rare. LLMs corral a similar randomness to statistically answer things correctly more often than not.
• ToValueFunfetti 2y ago
  A witness to a crime thinks that there were 6 shots fired; in fact there were only 2. They remember correctly the gender of the criminal, the color of their jacket, the street corner where it happened, and the time. There is no difference in their mind between the true memories and the false memory. I write six pieces of code that I believe have no bugs. One has an off-by-one error. I didn't have any different experience writing the buggy code than I did writing the functional code, and I must execute the code to understand that anything different occurred. Shall I conclude that myself and the witness were hallucinating when we got the right answers? That intelligence is not the thing that got us there?
  • sangnoir 2y ago
    People who are consistently unable to distinguish fiction from reality make terribly witnesses; if an obviously high crackhead would fare better than an LLM on the witness stand.
  • Gormo 2y ago
    > Shall I conclude that myself and the witness were hallucinating when we got the right answers? If you were recalling stored memories of experiences that were actual interactions with external reality, and some of those memories were subsequently corrupted, then no, you were not hallucinating. If you were engaging in a completely endogenous stochastic process to generate information independently of any interaction with external reality, then yes, you were hallucinating. > That intelligence is not the thing that got us there? It's not. In both cases, the information you are recalling is stored data generated by external input. The storage medium happens to be imperfect, however, and occasionally flips bits, so later reads might not exactly match what was written. But in neither case was the original data generated via a procedural algorithm independently of external input.
• poniexprsns 2y ago
  [dead]
• apantel 2y ago
  Here’s the issue: humans do the same thing: the brain builds up a model of the world but the model is not the world. It is a virtual approximation or interpretation based on training data: passed experiences, perceptions, etc. A human can tell you the sky is blue based on its model. So can any LLM. The sky is blue. So the output from both models is truthy.
  • Gormo 2y ago
    > A human can tell you the sky is blue based on its model. So can any LLM. The sky is blue. So the output from both models is truthy. But a human can also tell you that the sky is blue based looking at the sky, without engaging in any model-based inference. An LLM cannot do that, and can only rely on its model. Humans can engage in both empirical observation and stochastic inference. An LLM can only engage in stochastic inference. So while both can be truthy, only humans currently have the capacity to be truthful. It's also worth pointing out that even if human minds worked the same way as LLMs, our training data consists of an aggregation of exactly those empirical observation -- we are tokenizing and correlating our actual experiences of reality, and only subsequently representing the output of our inferences with words. The LLM, on the other hand, is trained only on that second-order data -- the words -- without having access to the much more thorough primary data that it represents.