3 ms·
From what I understand of the way LLMs work, they don't "know" when they're confabulating or not. All of the text they generate is arbitrary, albeit not entirel
by krapp 3y ago
From what I understand of the way LLMs work, they don't "know" when they're confabulating or not. All of the text they generate is arbitrary, albeit not entirely random. Whether or not any particular response is useful is a matter of human interpretation.
The problem is the tendency to assume LLMs behave the same way humans do. You know you're lying when you're lying. LLMs don't even have a concept of a "lie." Even though you can ask one what a lie is, and it responds with an accurate answer, that's still just an arbitrary statistically based response. It doesn't actually know.
- sfn42 3y agoIt's crazy to me how even in here there's such a huge amount of people who just don't understand. Even the article were discussing points this out > The implications are that LLMs do not perform reasoning over data in the way that most people conceive or desire. > There is no self-reflection of its information; it does not know what it knows and what it does not. The line between hallucination and truth is simply a probability factored by the prevalence of training data and post-training processes like fine-tuning It's not thinking, it's not conscious, it's just a mathematical function that's too complex for us to understand.
- anon373839 3y agoI think the best way to think of it is that it’s an estimation of the outputs of reasoning and knowledge. Of course this means that the models do need to model the reasoning process in some way, but it in no way implies that they model it in a way that’s rigorous or reliable.