4 ms·
Great post. Teaching the models to doubt, to say "I don't know"/"I'm unsure"/"I'm sure" is a nice way to make them much better.
by thomasboyer 1y ago
Great post. Teaching the models to doubt, to say "I don't know"/"I'm unsure"/"I'm sure" is a nice way to make them much better.
- more_corn 1y agoIt baffles me that this hasn’t been done yet. Saying I don’t know or I’m unsure is critical for anything that matters.
- ACCount37 1y agoMajor industry players were doing that for a while now. It's just hard to actually design training regimes that give LLMs better hallucination-avoidance capabilities. And it's easy to damage the hallucination-avoidance capabilities by training an LLM wrong. As OpenAI has demonstrated when they fried the o3 with RLVR that encouraged guesswork. That "SAT test incentivizes guesswork" example they give in the article is one they had to learn for themselves the hard way.
- meshugaas 1y agoLook at their stats though. If they did this, more than half of responses would end up as “I don’t know.” Nobody would use something that did that.
- skybrian 1y agoIt seems like it would train users to ask questions that it can actually answer. (They might also need some examples of what sort of questions to ask.)
- Jensson 1y agoMostly it would train users to not use their service and go to a service where the model outputs results they can copy paste to complete their assignment. So these companies cannot do this, they would hemorrhage too many users and companies cannot go against the profit incentives in practice.