24 ms·

Language models (the computation) "care" about making "correct" predictions (correct of course being determined by the data). This is why the simplest solution

Language models (the computation) "care" about making "correct" predictions (correct of course being determined by the data). This is why the simplest solution to reducing hallucinations is just making them more competent/better predictors (more neurons, more data). If the prediction can benefit from all the knowledge/reasoning it has accumulated from training, it'll tap that first. One of the biggest tells of hallucinations is if the model is saying something completely different each generation vs being consistent. What they just don't care about is communicating being out of distribution or making whack predictions. This becomes a problem for humans because they're perfectly fine making things up when the above fail. But by all accounts, they do learn to distinguish these things. The computation is very much aware when it is going way off base. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824

pdonis 3y ago
> Language models (the computation) "care" about making "correct" predictions Predictions of what? LLMs aren't predicting anything. They are, as the GP says, generating text that appears to be fluent and coherent (as long as we don't inquire into actual semantic relationships with anything else) in the context of the prompt.
- famouswaffles 3y ago
  They are quite literally trained to predict a distribution that could supply the next token given previous context. This is their primary objective.
  - pdonis 3y ago
    > They are quite literally trained to predict a distribution that could supply the next token given previous context. "Previous context" here means text. It does not mean "the actual world". Big difference.
    - famouswaffles 3y ago
      So ? That's not a big difference lol. The world can be sufficiently represented in text. and bolting in additional modalities to a transformer is trivial. Could just as well be images and audio also. You think you experience the "true" world ? You don't. You experience a slice of it that your brain often further fabricates at parts. To the birds that feel and sense electromagnetic waves intuitively to guide travels, your model of vision and direction is fundamentally incomplete/incorrect. No one gets to experience the real world.
      - pdonis 3y ago
        > That's not a big difference lol. You must be joking. You really think there's not a big difference between a corpus of text downloaded from the Internet and the entire actual world? Or even the sense-data that humans (or birds, for that matter) take in from the world? (And that doesn't even take into account the actions that humans, and birds, take in the world, and then compare the results with what their internal model predicts in order to update the model.) I don't even know how to respond to something that is so totally off base. > You think you experience the "true" world ? I have made no such claim. > To the birds that feel and sense electromagnetic waves intuitively to guide travels, your model of vision and direction is fundamentally incomplete/incorrect. Yes, but an LLM isn't a bird any more than it is a human. LLMs don't experience anything.
        famouswaffles 3y ago
        I'm not joking. Obviously there's a difference but it's not one of kind. There is a projection of the world in text. >Yes, but an LLM isn't a bird any more than it is a human. LLMs don't experience anything. Who is arguing whether LLMs are humans ? That's such an irrelevant point. The question is whether they are intelligent and understand which they are by any criteria that is actually testable. Nobody cares about building an artificial human. >LLMs don't experience anything. You don't know that

pdonis 3y ago

> Language models (the computation) "care" about making "correct" predictions Predictions of what? LLMs aren't predicting anything. They are, as the GP says, generating text that appears to be fluent and coherent (as long as we don't inquire into actual semantic relationships with anything else) in the context of the prompt.

famouswaffles 3y ago
They are quite literally trained to predict a distribution that could supply the next token given previous context. This is their primary objective.
- pdonis 3y ago
  > They are quite literally trained to predict a distribution that could supply the next token given previous context. "Previous context" here means text. It does not mean "the actual world". Big difference.
  - famouswaffles 3y ago
    So ? That's not a big difference lol. The world can be sufficiently represented in text. and bolting in additional modalities to a transformer is trivial. Could just as well be images and audio also. You think you experience the "true" world ? You don't. You experience a slice of it that your brain often further fabricates at parts. To the birds that feel and sense electromagnetic waves intuitively to guide travels, your model of vision and direction is fundamentally incomplete/incorrect. No one gets to experience the real world.
    - pdonis 3y ago
      > That's not a big difference lol. You must be joking. You really think there's not a big difference between a corpus of text downloaded from the Internet and the entire actual world? Or even the sense-data that humans (or birds, for that matter) take in from the world? (And that doesn't even take into account the actions that humans, and birds, take in the world, and then compare the results with what their internal model predicts in order to update the model.) I don't even know how to respond to something that is so totally off base. > You think you experience the "true" world ? I have made no such claim. > To the birds that feel and sense electromagnetic waves intuitively to guide travels, your model of vision and direction is fundamentally incomplete/incorrect. Yes, but an LLM isn't a bird any more than it is a human. LLMs don't experience anything.
      - famouswaffles 3y ago
        I'm not joking. Obviously there's a difference but it's not one of kind. There is a projection of the world in text. >Yes, but an LLM isn't a bird any more than it is a human. LLMs don't experience anything. Who is arguing whether LLMs are humans ? That's such an irrelevant point. The question is whether they are intelligent and understand which they are by any criteria that is actually testable. Nobody cares about building an artificial human. >LLMs don't experience anything. You don't know that