4 ms·
It doesn’t know what it doesn’t know.
by jeremyjh 1y ago
It doesn’t know what it doesn’t know.
- binarymax 1y agoWell sure. But maybe the token logprobs can be used to help give a confidence assessment.
- tyre 1y agoAnthropic has a great paper on exactly this! https://www.anthropic.com/research/language-models-mostly-know-what-they-know https://www.anthropic.com/research/language-models-mostly-kn... The best is its plummeting confidence when beginning the answer to “Why are you alive?” Big same, Claude.
- smt88 1y agoThat's not true for all types of questions. You've likely seen a model decline to answer a question that requires more recent training data than it has, for example.
- fallpeak 1y agoIt doesn't know that because it wasn't trained on any tasks that required it to develop that understanding. There's no fundamental reason an LLM couldn't learn "what it knows" in parallel with the things it knows, given a suitable reward function during training.