4 ms·

LLMs can't hallucinate. They generate the next most likely token in a sequence. Whether that sequence matches any kind of objective truth is orthogonal to how m

LLMs can't hallucinate. They generate the next most likely token in a sequence. Whether that sequence matches any kind of objective truth is orthogonal to how models work. I suppose depending on your point of view, LLMs either can't hallucinate, or that's all they can do.

CooCooCaCha 2y ago
Whenever someone takes issue with using the word “hallucinate” with LLMs I get the impression they’re trying to convince me that hallucination is good. Why do you care so much about this particular issue? And why can’t hallucination be something we can aim to improve?
ToValueFunfetti 2y ago
>Whether that sequence matches any kind of objective truth is orthogonal to how models work. Empirically, this cannot be true. If it were, it would be statistically shocking how often models coincidentally say true things. The training does not perfectly align the model with truth, but 'orthogonal' is off by a minimum of 45 degrees.
- viraptor 2y ago
  It matches the training data. Whether the training data matches truth (and whether it's correctly understood - sarcasm included) is a completely separate thing. > The training does not perfectly align the model with truth, but 'orthogonal' Nitpicky, but the more dimensions you have, the easier it is for almost everything to be orthogonal. (https://softwaredoug.com/blog/2022/12/26/surpries-at-hi-dimensions-orthoginality https://softwaredoug.com/blog/2022/12/26/surpries-at-hi-dime...) That's why averaging embeddings works.
  - timcobb 2y ago
    Isn't this the same thing that happens when you train a human on truths vs falsehoods?
  - ToValueFunfetti 2y ago
    I went to school to learn about the world and the overwhelming majority of that learning was from professors and textbooks. Whether the professors' beliefs and the textbooks' contents reflected the true properties of the world was a completely separate thing, entirely outside of my control. But I did come away with a better understanding of the world and few would say that education is orthogonal to that goal. If you add two vectors that don't have a truth component (ie. are orthogonal to the truth), the resulting vector should be no closer to the truth. If you start with random weights and perform some operation on them such that the new weights have a higher likelihood of producing true statements, the operation must not have been orthogonal to the truth. Am I wrong there?
    - viraptor 2y ago
      > But I did come away with a better understanding of the world and few would say that education is orthogonal to that goal. That's due to the reward function / environment. But even outside extremes like North Korea, lots of education environments value conformity over independent analysis.
      - ToValueFunfetti 2y ago
        Certainly an AI trained on North Korean data would emerge with some very suspect beliefs regarding Kim Jong-Un. My point is just that aligning something with training data is aligning it with truth, to the degree that the training data is true and regardless of why it is true. educate(me, truth) can hardly be called orthogonal to the truth, even if the 'educate' and 'me' terms do nothing to prevent educate(me, falsehood).

CooCooCaCha 2y ago

Whenever someone takes issue with using the word “hallucinate” with LLMs I get the impression they’re trying to convince me that hallucination is good. Why do you care so much about this particular issue? And why can’t hallucination be something we can aim to improve?

ToValueFunfetti 2y ago

>Whether that sequence matches any kind of objective truth is orthogonal to how models work. Empirically, this cannot be true. If it were, it would be statistically shocking how often models coincidentally say true things. The training does not perfectly align the model with truth, but 'orthogonal' is off by a minimum of 45 degrees.

viraptor 2y ago
It matches the training data. Whether the training data matches truth (and whether it's correctly understood - sarcasm included) is a completely separate thing. > The training does not perfectly align the model with truth, but 'orthogonal' Nitpicky, but the more dimensions you have, the easier it is for almost everything to be orthogonal. (https://softwaredoug.com/blog/2022/12/26/surpries-at-hi-dimensions-orthoginality https://softwaredoug.com/blog/2022/12/26/surpries-at-hi-dime...) That's why averaging embeddings works.
- timcobb 2y ago
  Isn't this the same thing that happens when you train a human on truths vs falsehoods?
- ToValueFunfetti 2y ago
  I went to school to learn about the world and the overwhelming majority of that learning was from professors and textbooks. Whether the professors' beliefs and the textbooks' contents reflected the true properties of the world was a completely separate thing, entirely outside of my control. But I did come away with a better understanding of the world and few would say that education is orthogonal to that goal. If you add two vectors that don't have a truth component (ie. are orthogonal to the truth), the resulting vector should be no closer to the truth. If you start with random weights and perform some operation on them such that the new weights have a higher likelihood of producing true statements, the operation must not have been orthogonal to the truth. Am I wrong there?
  - viraptor 2y ago
    > But I did come away with a better understanding of the world and few would say that education is orthogonal to that goal. That's due to the reward function / environment. But even outside extremes like North Korea, lots of education environments value conformity over independent analysis.
    - ToValueFunfetti 2y ago
      Certainly an AI trained on North Korean data would emerge with some very suspect beliefs regarding Kim Jong-Un. My point is just that aligning something with training data is aligning it with truth, to the degree that the training data is true and regardless of why it is true. educate(me, truth) can hardly be called orthogonal to the truth, even if the 'educate' and 'me' terms do nothing to prevent educate(me, falsehood).