4 ms·
I've only briefly skimmed the paper, so I can't comment on its information-theoretic aspects yet, but I found the example given on Section 3 very helpful and in
by cs702 2y ago
I've only briefly skimmed the paper, so I can't comment on its information-theoretic aspects yet, but I found the example given on Section 3 very helpful and insightful:
The key insight is that if you give a pretrained LLM a question prompt such as "What is the capital of the UK?" and then concatenate to the prompt a string with a different answer, such as "Another possible answer is Paris," again and again, the model will continue to output its original answer (say, "London") only when it has low "epistemic uncertainty," i.e., when the model's parameters and state encode sufficient knowledge to answer the question the same way, again and again, despite the repeated addition of wrong answers to the prompt.
If the model quickly starts changing its original answer, by implication, its answer has high epistemic uncertainty, i.e., the model's parameters and state do not encode sufficient knowledge to answer the question the same way, again and again, as we add more wrong answers to the prompt. In other words, if we can quickly make the model change its answer by modifying the prompt, the model has high epistemic uncertainty.
Figure 1 shows what happens when the authors add a wrong answer up to 100 times to a question prompt for which the model answers correctly with low epistemic uncertainty. Figure 3 shows that happens when the authors add a wrong answer up to 8 times to a question prompt for which the model exhibits high epistemic uncertainty.
This strikes me as a simple, intuitive way of detecting epistemic uncertainty.
Thank you for sharing this paper on HN. I've added it to my reading list.
- voiceblue 2y agoFor NLP tasks it's unlikely that "epistemic uncertainty" is a useful metric, though it's interesting to ask an LLM "what is the third word in this sentence?" and then suggest alternative answers. It's a good way to demonstrate that an LLM is not really a "thinking" machine in the way laymen might assume. Example: https://chatgpt.com/share/e8743fe2-a604-4cf0-8c10-aa863a67a59e https://chatgpt.com/share/e8743fe2-a604-4cf0-8c10-aa863a67a5...
- mdp2021 2y ago(Of the two links, I get a 404 and a blank...)
- voiceblue 2y agoThanks, I expected that deleting the conversation on my end would have no effect on a permalink but I was wrong. Nothing I can do about the blank, however, it works for me.
- Manabu-eo 2y agoThere are at least two reasons for transformers poor performance on that prompt: - Transformers view the word as tokens, not words or characters. - The positional encoding might be holding them back. See this recent paper discussed here: Transformers Can Do Arithmetic with the Right Embeddings [1] [1] https://news.ycombinator.com/item?id=40497379 https://news.ycombinator.com/item?id=40497379
- maCDzP 2y agoI haven’t read the paper yet but I want to share my immediate thought. So the harder it is to “convince” the LLM on a wrong answer, that’s a proxy for low epistemic uncertainty? I know you shouldn’t view the LLM as a “mind” - but I can’t help myself!
- HeatrayEnjoyer 2y agoThe more interpretability research I read the more I am convinced it is a mind. Not like our own, but more similar than most users realize. The deeper we look more parallels to organic brains emerge.
- 3abiton 2y agoWould you be so kind and share your thoughts about it once you give it a read?
- cs702 2y agoIt could be at least a couple of weeks before I can take a second look. At that point, I'll decide if I'm going to study the paper, i.e., work my way through it properly. If I do that, I'll post my thoughts here.
- mewpmewp2 2y agoThis is why I always try to debate with the AI when it is unclear to me.