4 ms·
This counts only incorrect answers though. A model can get 0% hallucination rate just by refusing to answer all questions.
by jampekka 5mo ago
This counts only incorrect answers though. A model can get 0% hallucination rate just by refusing to answer all questions.
- ffsm8 5mo agoIsn't that precisely the reason why we introduced the term hallucination? Because llms have historically always made up bullshit of they cannot answer directly... If they now nailed this to maybe the model not respond instead of responding incorrectly, then a lot of previously unusable usecases would become feasible. So I feel like that's exactly the right metric and the way to track it wrt hallucinations.
- doublescoop 5mo agoI had a buddy in high school that was notorious for doing the same thing. (He's now a senior director at a Big 4 consultancy. :) )
- rrgok 5mo agoDo you mind expanding a little more?
- alfiedotwtf 5mo agoThey had a buddy who used to lie a lot when they were younger… now they get paid for it
- akoboldfrying 5mo agoThe point is that it's not a useful metric on its own. For example, redirecting from /dev/null also achieves a zero hallucination rate. We want the hallucination rate to decrease while the overall answer rate of queries remains sufficiently high. For more specifics, look into ROC and AUC.
- speed_spread 5mo agoYes. A model that can answer "I don't know" would be much more trustable than the current used car salesman we have now.
- jorvi 5mo agoIts very annoying this has been in the capability of models since the very beginning. It could check how probable its token values are and if those fall below a certain threshold either say "I don't know", or output the most probable (well, more like least improbable) tokens but give a very clear, very strong warning that it is a shot in the dark and likely to contain hallucinations. But no, Google and OpenAI would rather always have an answer ready and tell you to mix glue into your pizza toppings :)
- nomel 5mo agoYeah, I never understood why the top n statistics weren't included in the chat interfaces, to color the text!
- tokenscoper 5mo agoI don't have much to add other than this observation that we seem to have moved away from eating one small rock per day for nutritional value, and adding gasoline in spaghetti. The glue on pizza reference brought back memories :)
- miki123211 5mo agoIt can't, because top n isn't always reliable. Hallucination detection is an open problem. If it were that simple, people would indeed "just" do it. Basically the problem is that LLMs aren't trained on things they don't know; an alternative way of saying this is that they're not trained on things they're not trained on, which is obviously true. When you RL a model and it answers incorrectly, you don't teach it to answer "I don't know", you teach it to answer correctly instead. This makes it very hard for it to realize when it doesn't know things.
- chengyongru 5mo ago
- jug 5mo agoI think that's what the Omniscience Index is for: https://artificialanalysis.ai/evaluations/omniscience#aa-omniscience-index-results https://artificialanalysis.ai/evaluations/omniscience#aa-omn... It rewards correct answers and penalizes hallucinations, and finally no reward for refusing to answer. It's interesting just how poorly some popular Chinese models fare in this regard, like GLM 5.1 or DeepSeek 4 Pro. Gemini 3.x has truly remarkable knowledge given how it leads in this benchmark despite being (quite a bit) more prone to hallucinate than Claude Opus.
- aicantdeny 5mo ago> by refusing to answer all questions. Cool, precisely the thing other AI is too stupid to do when they don't have the necessary knowledge.
- Balinares 5mo agoYes, that's in fact precisely the desired behavior when a model doesn't know the answer.