6 ms·

The LLM AI technology generation is optimized to be fluently conversational and not to be factually correct all the time. 1) Hallucinations often appear becaus

• thfuran 3y ago
  Did you read the article?
  • vinceguidry 3y ago
    skims article Nope, they summarized it without reading it. Useful enough for those who want to keep a finger to the pulse of AI without a huge time investment. God save me from having to read every article on HN.
  • happytiger 3y ago
    I did. I’m aware. But I thought the comment worth writing anyways. I’m sorry it wasn’t enough of a response to the article for you, but that wasn’t my intention in writing it.
• teaearlgraycold 3y ago
  > LLMs have no understanding of the underlying reality Not true. They are slowly gaining an understanding of reality by reverse engineering the relationships built into human languages. The only reason LLMs are getting better is because they are better modeling the world. At some point the only way to improve token prediction is to gain an understanding of the world.
  • pdonis 3y ago
    > They are slowly gaining an understanding of reality by reverse engineering the relationships built into human languages. Perhaps the people building LLMs are doing this, but the LLMs themselves are not. LLMs are just generating text. They aren't "reverse engineering" anything. > The only reason LLMs are getting better is because they are better modeling the world. No, they are getting better at generating text that seems fluent and coherent, as long as you don't inquire into any actual semantic relationships with the world. LLMs can't model the world because they don't even have a concept of "the world". All they have is text.
    • teaearlgraycold 3y ago
      Here's a bit from an episode of This American Life that I think is relevant, cut into a post on my blog: https://danangell.com/blog/posts/gpt-understands/ https://danangell.com/blog/posts/gpt-understands/ > The idea is that back when GPT-4 was being trained for it to really consistently get the next word correct, to do that reliably, it had to do more than just bullshit. It had to do more than guess based on patterns. To get the next word right, it had to truly understand the words coming before it. Imagine if you were training a transformer to predict numbers in a sequence that came from a sine function. Eventually it would get a pretty good test score and you'd find that internally it had built a decent replication of the sine function. Now do the same with English text. Throw enough data and computation at it and eventually it'll model a rough understanding of gravity and the shapes of objects, as seen in the different answers between GPT-3.5 and GPT-4.
      • thfuran 3y ago
        Does an analog circuit understand differential equations?
        • teaearlgraycold 3y ago
          I think it's not really interesting to say so, but by the same definition it would be fair to describe an analog circuit that models some system as understanding the system in some capacity. It's more interesting when the thing doing the understanding is a massive system that has both mastered English and other domains and then incidentally modeled some systems English can describe. An analog circuit that models artillery trajectories "understands" one kinematic equation. But that's kind of obvious and mundane. Understanding just one equation barely registers as profound and without that understanding existing in a larger context the word barely seems applicable.
• famouswaffles 3y ago
  Language models (the computation) "care" about making "correct" predictions (correct of course being determined by the data). This is why the simplest solution to reducing hallucinations is just making them more competent/better predictors (more neurons, more data). If the prediction can benefit from all the knowledge/reasoning it has accumulated from training, it'll tap that first. One of the biggest tells of hallucinations is if the model is saying something completely different each generation vs being consistent. What they just don't care about is communicating being out of distribution or making whack predictions. This becomes a problem for humans because they're perfectly fine making things up when the above fail. But by all accounts, they do learn to distinguish these things. The computation is very much aware when it is going way off base. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824
  • pdonis 3y ago
    > Language models (the computation) "care" about making "correct" predictions Predictions of what? LLMs aren't predicting anything. They are, as the GP says, generating text that appears to be fluent and coherent (as long as we don't inquire into actual semantic relationships with anything else) in the context of the prompt.
    • famouswaffles 3y ago
      They are quite literally trained to predict a distribution that could supply the next token given previous context. This is their primary objective.
      • 3y ago
• jshen 3y ago
  > "The LLM AI technology generation is optimized to be fluently conversational and not to be factually correct all the time." I always find this point a bit odd because humans aren't "optimized" to be correct either. > "LLMs have no understanding of the underlying reality" I struggle with this one because I see both sides of it. I was making a prompt the other day and gave a CSV file as an input and told the LLM if could only answer with values from one column and it did exactly as I asked. It's hard for me to see things like that and not believe it has an understanding at some level.
  • km3r 3y ago
    There is a difference between general AI and generative AI. Humans can be creative and create vast fictional worlds with made up technology and magic. They can also optimize for facts and pass a BAR exam. LLM like have some form of understanding of the world, not sure if its anywhere near comparable to our understanding though. But they are still not general enough to really focus on facts. They can get close but the nature of statistics means there isn't hard checks to truth.
    • happytiger 3y ago
      I think the next step logically is to see how the human brain functions. We have a similar model where certain parts of our brain are hopelessly useless by themselves as they are so overly specialized in one thing, but work in harmony to create network effectiveness. And we only just [mapped it for the first time](https://phys.org/news/2023-10-scientists-generate-single-cell-atlas-primate.html https://phys.org/news/2023-10-scientists-generate-single-cel...) so now much of what we are learning can potentially be applied to our “digital twins” in coming years. I imagine a lot of progress with AI will involve similar networks, with governance providing evolutionary paths and guidance for multiple concurrent goals. There are also lots of issues with training data and memory limits that make LLMs weak at continuity. Holes or weaknesses in the training data might “come through” in the behavior of hallucination. Where do you think general AI is going right now that we should be looking at? I’m overwhelmed by how large this field has become in the last five years and am always interested in what others know.
  • 3y ago
• 98codes 3y ago
  When a human reads a bunch of stuff on the internet and regurgitates it as fact, we don't say they hallucinate, we say that they're wrong. Why we baby AI on this front, I have no idea.
  • Avshalom 3y ago
    Because if "AI" is constantly "wrong" or "lying" it'd be bad for the businesses trying to sell it. So there's this marketing push to frame that recognition of constant obvious failure as anthropomorphizing the AI. Hallucinating on the other hand feels way more abstract. We speak of humans hallucinating answers rarely enough that they can use it for AI with a straight face. Heck, when we do talk about humans hallucinating it's often-as-not in the context of mind expanding experiences; maybe AI hallucinations are even a good thing!
    • happytiger 3y ago
      We actually have a core personality characteristic in human psychology for just this thing: neuroticism. It may be part of complex networks and intelligence, and humans may have a lot of it happening all the time but have ways of compensation or suppression of neurotic or hallucinating thoughts. We have some intense “mental illnesses” that look a lot like what AI is presenting, where it can’t tell what is going on and fills in the gaps (schizophrenia maybe) or uses behavioral patterns to compensate for memory or intelligence inadequacy. Obviously straight conjecturing here.
• pdonis 3y ago
  > All these criticisms of hallucinations are missing the point. Not if people need to be reminded that, as you say, LLMs are not designed to give reliable answers. Many, many people appear to believe that they are, since they rely on the answers in all kinds of contexts. For example, in past HN threads on LLMs, I have seen people say that they rely on LLM-generated code.
  • happytiger 3y ago
    In the land of the blind the one eyed man is king? ;)
  • gaganyaan 3y ago
    I rely on code written by AI and verified by me, particularly when the code is finicky or tedious but verification is simple. LLMs are more reliable than you're trying to assert, but blindly copy/pasting AI-generated code is a recipe for pain, just the same as blindly copy/pasting human-generated code from SO.
    • pdonis 3y ago
      > I rely on code written by AI and verified by me The "verified by me" part is crucial, though, yes? I'm talking about people, whom I have seen posting here on HN in other threads on this topic, who leave out the "verify" step and just rely on the LLM code as it comes.
      • gaganyaan 3y ago
        Yes, verification (or trust) is crucial. Those posts may have elided the verification process they did, though. Unless we look at a specific post it'd be hard to discuss it. To be more specific, I trust GPT-4 to handle tasks like "Rewrite this C++ struct into Rust". I still verify that it's correct as part of a normal code review process, but I trust GPT-4 in that scenario at least as much as I would a junior dev.
• hatthew 3y ago
  > LLMs have no understanding of the underlying reality that language describes. LLMs absolutely have concepts that extend beyond just words. As early as 2008 (back when there were no large language models, only "regular" language models), we've been able to demonstrate things that seem to me like the model is learning abstract concepts. For a classic example, see Linguistic Regularities in Continuous Space Word Representations [0], a 2013 paper that talks about a model with a latent space where one can take the vector representations of the words "king", "queen", "man", and "woman" and literally perform the arithmetic `king - (man - woman) ≈ queen`. To me, this clearly demonstrates that the model "understands" the concept of gender, represented by a vector in the model's latent space. The set of numbers that you get from the subtraction `man - woman` represent the concept of the difference between the male and female genders, without needing a specific word tied to that representation of the concept. It's debatable whether or not that counts as "understanding", but that's more of a semantic debate about what it means to understand, not a debate about the model's internal knowledge of the world and ability to do things with that knowledge. 0: https://aclanthology.org/N13-1090.pdf https://aclanthology.org/N13-1090.pdf
  • yladiz 3y ago
    Are you sure the model in the paper linked and LLMs function similarly or identically in this regard?
    • hatthew 3y ago
      No, but IMO an assertion that they have no understanding of the world has less supporting evidence than an assertion that they do have some level of understanding.
  • happytiger 3y ago
    I would suggest that example is more likely evidence of semantic associations provided by underlying taxonomy based on input training data. There is so much contextual information in language hierarchy and associated characteristics and there is nothing stopping a LLM from using multiple recombinant factors for generation. I don’t mean to discount your excellent comment. I mean obviously these models are so complex already that their black box nature makes it difficult to derive conclusive results, but Arkham’s razor would imply that the simplest explanation — that is it recombined training data creating a heirarchy with multiple factors that provides context for the “correct” answer — that is more likely correct. In other words a LlM can confer multiple related associations to output based on training data, and that is the most likely explanation for that behavior. Just look at the poor performance with GPT-3 and the underlying data quality issues the Alpaca dataset team has discussed. I’m no expert, and don’t mean to counter you as much as contribute my own limited thinking for discussion here. Cheers!
• IIAOPSW 3y ago
  In other words, LLM's are, in the technical sense defined by Harry Frankfurt, bullshitters. The essence of bullshit (as he defines it) is that it is neither truth nor lie but rather unconcerned with veracity altogether. A bullshitter wants to appear smart, or convince others to agree with them, or some other conversational objective. Bullshit statements may or may not ultimately turn out to be true, but the thing that makes them bullshit is that the bullshitter didn't actually know when they said it. Hence I propose a far more succinct term to describe the phenomena at hand. A far more accurate terminology than the presently vogue "hallucinating": AB. Artificial Bullshit.
  • invig 3y ago
    Isn’t the same thing true of any human who isn’t doing scientific research? Like how much work to get to veracity is actually going on out there?
    • deleted 3y ago
      [deleted]
    • IIAOPSW 3y ago
      “One of the most salient features of our culture is that there is so much bullshit.” Literally the cold opening of "On Bullshit".