A language model is just that.. it predicts probabilities of words given some prefix. Of course such a thing cannot hallucinate. What is confabulating is the hu

A language model is just that.. it predicts probabilities of words given some prefix. Of course such a thing cannot hallucinate. What is confabulating is the human imposed process of selecting a random word weighted on the probability distribution returned. Obviously it's going to cause a problem. Imagine communicating with someone and they ask a yes or no question and you attempt to tell them the truth which is that you are 70 percent yes and 30 percent no. And then they apply a randomized picking algorithm that weights those responses and when it comes back as 'no', they go and tell everyone you lied. Unless we make word output an actual part of LMs we can't see they are lying or not lying. They only produce probabilities of word outputs based on the context.