5 ms·

> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afte

> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.

gwerbin 2mo ago
If you watch the thinking traces of just about any modern LLM, you might be surprised at how much "uncertainty" is in there. Weak models with no thinking limits vacillate back-and-forth back-and-forth on a topic for potentially thousands of tokens before gradually spiraling towards some kind of an answer. Which makes it all the more interesting that "I don't know" is so rarely the final prediction, even with so much waffling in the chain of thought. Until the big labs decide to start adding synthetic "I don't know" outcomes to their data sets, I've been thinking that the best way to evaluate uncertainty is to have a separate LLM monitoring the conversation and asking it to classify if the agent is overstating its confidence. On the other hand I've also noticed that most models will tell you they don't know something if you specifically include it in the prompt, eg "if you don't know the answer, just say so" and/or "be clear about any gaps in your knowledge that would reduce the confidence of your response" etc. but even with the big frontier models I have noticed some quality degradation if I throw too many instructions into the system prompt. I have a little more faith in harness-level engineering than in praying to the token generation gods. That said, there is a completely different form of "uncertainty" in which the LLM tends to place very high trust in its own prior outputs as well as user provided inputs. Again if you look at the thinking traces, these models will try very very hard to rationalize the inputs they are given, falling back to the possibility of user error only after working through several alternative possibilities, maybe even investigating data or source code in the process. And if your context is big enough, the model might just completely miss when pieces of information conflict.

gwerbin 2mo ago

If you watch the thinking traces of just about any modern LLM, you might be surprised at how much "uncertainty" is in there. Weak models with no thinking limits vacillate back-and-forth back-and-forth on a topic for potentially thousands of tokens before gradually spiraling towards some kind of an answer. Which makes it all the more interesting that "I don't know" is so rarely the final prediction, even with so much waffling in the chain of thought. Until the big labs decide to start adding synthetic "I don't know" outcomes to their data sets, I've been thinking that the best way to evaluate uncertainty is to have a separate LLM monitoring the conversation and asking it to classify if the agent is overstating its confidence. On the other hand I've also noticed that most models will tell you they don't know something if you specifically include it in the prompt, eg "if you don't know the answer, just say so" and/or "be clear about any gaps in your knowledge that would reduce the confidence of your response" etc. but even with the big frontier models I have noticed some quality degradation if I throw too many instructions into the system prompt. I have a little more faith in harness-level engineering than in praying to the token generation gods. That said, there is a completely different form of "uncertainty" in which the LLM tends to place very high trust in its own prior outputs as well as user provided inputs. Again if you look at the thinking traces, these models will try very very hard to rationalize the inputs they are given, falling back to the possibility of user error only after working through several alternative possibilities, maybe even investigating data or source code in the process. And if your context is big enough, the model might just completely miss when pieces of information conflict.