3 ms·

I think this article pushes the premise farther than is reasonable. The best anyone expects from an LLM "truth vector" is that it would encode the model's beli

I think this article pushes the premise farther than is reasonable. The best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. Of course a perfect truth oracle is impossible.

samlinnfer 2mo ago
Ah but is the model's belief of the statements 1. complete, 2. consistent, 3. decidable?
- Legend2440 2mo ago
  I think this is a category error. Those are properties of systems of logical axioms. But LLMs are not logical; they are statistical. It can only 'believe' a statement is true in the Bayesian sense, where the statement agrees with the priors.
TZubiri 2mo ago
If anything, the whole vector space is the LLM's truth.
- zarzavat 2mo ago
  LLMs are not optimized only for truth they are optimized for a more complicated objective that includes e.g. humans liking their output. It is a universal truth that to get humans to like you, you have to lie to them.
  - NitpickLawyer 2mo ago
    > It is a universal truth that to get humans to like you, you have to lie to them. Thus said HAL9000
  - elendilm 2mo ago
    <It is a universal truth that to get humans to like you, you have to lie to them.> You don't represent all of us buddy. Some of us love truth above all else.
    - ngruhn 2mo ago
      Not many though
    - lelanthran 2mo ago
      > You don't represent all of us buddy. Some of us love truth above all else. So? Some people get off on pain, doesn't make the statement "People try to avoid pain" false.
  - TeMPOraL 2mo ago
    Yes. Much like our own communication. Thus is the difference between a research paper and the poem. People like the former for objectivity, the latter for beauty.
Terr_ 2mo ago
> The best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. I think even that's too-optimistic: The LLM is a document-extender, so its "belief" is whether a token seems like it would statistically fit-next in a partial document, based on prior documents. This is usually not the kind of analytic truth we're interested in, and we've already figured out how to constantly extract it. If we peek at vectors and weights, we'll we'll probably end up measuring the moods and styles for whatever tokens are about to get emitted next, whether that's dialogue for a fictional character (of various kinds), a narrator, or an impersonal memo conclusion paragraph. We'll be measuring "earnestness and conviction", on the same level as "loquaciousness" or "pleading" or "talking like a pirate." So is Truthiness [0] what we really want? Probably not. If our document described the character as Yoda, then The Force connecting all existence ends up truthy. Using "a really gullible person" can repeat anything you supply as truthy. Even if we set things up as "a respected encyclopedia article" or "a relentlessly logical super-genius", we're really changing the influence mix of styles and biases, rather than creating a logical mind independent of text inside the LLM. [0] https://en.wikipedia.org/wiki/Truthiness https://en.wikipedia.org/wiki/Truthiness
- FeepingCreature 2mo ago
  I half-disagree: this is exactly the kind of analytic truth humans are usually interested in. "Truth" as perceived by humans is based on a massive system of prefiltering, narrowing, preprocessing and situational awareness. A drop of water falls on your hand. Is it raining? Depends. Are you painting a watercolor picture outside? Then probably yes. Or are you going to the store? Then probably no. So truth is inseparable in practice from usecase. Is a whale a fish? I don't know, are you a geneticist or a poet? It's all mood and usecase. I'm not convinced there's any difference in kind between the LLM's speaker-selection and a human's choice of research field. That is to say, it's not that I think you're wrong, it's that there is no other pursuit of truth than what you describe. If anything, the LLM's pursuit of next-token prediction is unusually honest for a truth-seeker.
- TeMPOraL 2mo ago
  Worth remembering what the goal function behind the next token prediction is. It's what makes it go beyond moods and styles, and work with concepts of fact the same way we do.
mike_hearn 2mo ago
Not only that, but nobody really cares about whether truth vectors can work correctly under carefully constructed paradox edge cases. Well, maybe mathematicians do, but nobody else.
- TeMPOraL 2mo ago
  Mathematicians, philosophers, cognitive scientists, and others. The latent space encoded in the weights of a model is itself an object of study, just as interesting as the model itself.
  - cyanydeez 2mo ago
    someone on HN about a year ago was trying to argue that an LLM is not a cultural artifact for an anthropologist to study. There's definitely people around that have no idea about the "soft" sciences and how an LLM while constructed from a "hard" science is almost entirely a "soft" science object.
lesostep 2mo ago
As I understand it, this argument holds exactly the same for any type of truth: both objective and subjective (assuming subject is logically sound). Whatever vector we choose, there always would be contradictions. And while people also have internal conflicts, they are capable of adapting their belief system, while adjusting a vector would just give an LLM new self-contradictory classifier.

samlinnfer 2mo ago

Ah but is the model's belief of the statements 1. complete, 2. consistent, 3. decidable?

Legend2440 2mo ago
I think this is a category error. Those are properties of systems of logical axioms. But LLMs are not logical; they are statistical. It can only 'believe' a statement is true in the Bayesian sense, where the statement agrees with the priors.

TZubiri 2mo ago

If anything, the whole vector space is the LLM's truth.

zarzavat 2mo ago
LLMs are not optimized only for truth they are optimized for a more complicated objective that includes e.g. humans liking their output. It is a universal truth that to get humans to like you, you have to lie to them.
- NitpickLawyer 2mo ago
  > It is a universal truth that to get humans to like you, you have to lie to them. Thus said HAL9000
- elendilm 2mo ago
  <It is a universal truth that to get humans to like you, you have to lie to them.> You don't represent all of us buddy. Some of us love truth above all else.
  - ngruhn 2mo ago
    Not many though
  - lelanthran 2mo ago
    > You don't represent all of us buddy. Some of us love truth above all else. So? Some people get off on pain, doesn't make the statement "People try to avoid pain" false.
- TeMPOraL 2mo ago
  Yes. Much like our own communication. Thus is the difference between a research paper and the poem. People like the former for objectivity, the latter for beauty.

Terr_ 2mo ago

> The best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. I think even that's too-optimistic: The LLM is a document-extender, so its "belief" is whether a token seems like it would statistically fit-next in a partial document, based on prior documents. This is usually not the kind of analytic truth we're interested in, and we've already figured out how to constantly extract it. If we peek at vectors and weights, we'll we'll probably end up measuring the moods and styles for whatever tokens are about to get emitted next, whether that's dialogue for a fictional character (of various kinds), a narrator, or an impersonal memo conclusion paragraph. We'll be measuring "earnestness and conviction", on the same level as "loquaciousness" or "pleading" or "talking like a pirate." So is Truthiness [0] what we really want? Probably not. If our document described the character as Yoda, then The Force connecting all existence ends up truthy. Using "a really gullible person" can repeat anything you supply as truthy. Even if we set things up as "a respected encyclopedia article" or "a relentlessly logical super-genius", we're really changing the influence mix of styles and biases, rather than creating a logical mind independent of text inside the LLM. [0] https://en.wikipedia.org/wiki/Truthiness https://en.wikipedia.org/wiki/Truthiness

FeepingCreature 2mo ago
I half-disagree: this is exactly the kind of analytic truth humans are usually interested in. "Truth" as perceived by humans is based on a massive system of prefiltering, narrowing, preprocessing and situational awareness. A drop of water falls on your hand. Is it raining? Depends. Are you painting a watercolor picture outside? Then probably yes. Or are you going to the store? Then probably no. So truth is inseparable in practice from usecase. Is a whale a fish? I don't know, are you a geneticist or a poet? It's all mood and usecase. I'm not convinced there's any difference in kind between the LLM's speaker-selection and a human's choice of research field. That is to say, it's not that I think you're wrong, it's that there is no other pursuit of truth than what you describe. If anything, the LLM's pursuit of next-token prediction is unusually honest for a truth-seeker.
TeMPOraL 2mo ago
Worth remembering what the goal function behind the next token prediction is. It's what makes it go beyond moods and styles, and work with concepts of fact the same way we do.

mike_hearn 2mo ago

Not only that, but nobody really cares about whether truth vectors can work correctly under carefully constructed paradox edge cases. Well, maybe mathematicians do, but nobody else.

TeMPOraL 2mo ago
Mathematicians, philosophers, cognitive scientists, and others. The latent space encoded in the weights of a model is itself an object of study, just as interesting as the model itself.
- cyanydeez 2mo ago
  someone on HN about a year ago was trying to argue that an LLM is not a cultural artifact for an anthropologist to study. There's definitely people around that have no idea about the "soft" sciences and how an LLM while constructed from a "hard" science is almost entirely a "soft" science object.

lesostep 2mo ago

As I understand it, this argument holds exactly the same for any type of truth: both objective and subjective (assuming subject is logically sound). Whatever vector we choose, there always would be contradictions. And while people also have internal conflicts, they are capable of adapting their belief system, while adjusting a vector would just give an LLM new self-contradictory classifier.