5 ms·

Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the pr

Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. I'd also hesitate to attribute this difference in hallucination rates purely to model size. Yes, GLM-5.2 hallucinates much less frequently than DeepSeek-V4 Pro with twice as many parameters, but DeepSeek-V4 Flash is less than half the size of GLM-5.2 and tops the AA-Omniscience hallucination index. Opus 4.8, which is likely larger than DeepSeek-V4 Pro, has a 36% hallucination rate on the index, above GLM-5.2's 28%, but way below the DeepSeek numbers. Opus also has a 47% accuracy rate vs GLM-5.2's 25%. If you use these numbers to calculate the absolute hallucination rate (i.e., the number of hallucinated responses divided by the total number of responses), you get 19% for Opus and 21% for GLM-5.2. So yes, all else equal larger models may be more prone to hallucination in scenarios where they don't know the answer, but there are a lot of other factors that affect hallucination rates, and it's not totally clear that this is the main metric that's worth tracking.

in-silico 4mo ago
Additionally, maybe it's easier for a model to realize that it doesn't know the answer when the question is easier. If Opus gets all but the hardest questions right, it might have a higher hallucination rate because the questions it gets wrong are the questions where verification or hallucination detection are the most difficult
grayhatter 4mo ago
> Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. Do you have a cite for this? If a human makes up some bullshit lie, I wouldn't accuse them of making it up only if they actually knew the correct answer. If you don't know, the only correct answer is I don't know. Any other answer is made up bullshit. Why is it only a hallucination if and only if the LLM contains the answer? If you make something up it's still wrong. It shouldn't matter if you could give the correct answer. You didn't, and instead invented some bullshit instead? Follow up question, how can I apply this rule set to the next test I have to take? I'd love to be able to use "I didn't know" as the excuse for why I made something up. edit: > and it's not totally clear that this is the main metric that's worth tracking. I don't know, the rate at which some model is willing to make up something feels useful. If the argument I see repeated on HN so much is that it's impossible to completely get rid of hallucinations; being able to choose a model that's less likely to invent some lie seems like a positive trait, no? Either way, I'm happy to agree that a restrictive definition, where a lie doesn't count as a hallucination iff the model doesn't know the answer feels strictly, infinitely less useful than an exact error rate. What percentage of emitted tokens are misleading would be useful for me. Anyone know any group that's attempted to quantify the global error rate?
- sgc 4mo ago
  Since models just output the the most probable tokens and you can never accuse them of doing anything other than making it all up, I would like to see these tests run with a prompt that attempts to mitigate hallucination and finishes with something like: "Telling me that you don't have the relevant information or that the task is impossible is extremely useful to me and a valid answer", and see how much that changes the scoring - as well as the usefulness of the answers. There are so many skills like context7 that can be tweaked to improve these results as well. In other words, you shouldn't choose the model that hallucinates the least without detailed prompting, since a well-crafted agents.md clause should go a long way to improving output, and almost certainly the top scoring order will be different. To the point that I don't find this type of raw comparison useful beyond maybe 'make sure you test that one with more explicit prompts'.
  - grayhatter 4mo ago
    > In other words, you shouldn't choose the model that hallucinates the least without detailed prompting You're prompting it wrong is quickly becoming the new, you're holding it wrong. It's wild how willing software engineers are to blame the user when the actual problem is their own defective design. Ideally we all, as an industry, will stop accepting this as reasonable excuse for the demonstrated incompetence
    - ordersofmag 4mo ago
      It's not that you're prompting it wrong. It's that you're judging the output against a standard (human intelligence) that just isn't relevant--no matter how much we want it to be and no matter how much the fluency of the output tricks us into thinking there's a human-like mind behind it. Now granted, if the boat salesmen were pushing hard on the idea that the boat would fly and even put little wings on the side and I bought the boat I might get really angry when I found out that it didn't fly. And I might angrily storm into the salesroom yelling about how the design is defective. But if someone pointed out 'hey, it's a boat perhaps you should stick to sailing around in it and stop getting your undies in a bundle about it not flying' the correct response is probably to take a closer look, ignore the salesmen, and cruise around the lake. LLM's are quite handy at some things and have some weird limits. Learn the limits, enjoy your time at sea.
reinitctxoffset 4mo ago
Hallucination should be called "failure to ground". Something about the cost model of US near frontier has the cattle prod out whenever a model is uncertain but thrashes on whether to search. Search flinch is roughly all hallucination. I don't even wait for the model's turn, if there's a man page or Hoogle hit, stuff the last prefix cache cut point. You come out ahead.
sudosysgen 4mo ago
This is missing a common failure mode, which is information past the knowledge cutoff. If you need info past that time they'll fail no matter how big or small the model is, so the hallucination rate can matter independently of the knowledge base. If all use-cases had a uniform risk of falling out of support, this would be a valid argument, but since it's often the case that a datapoint is guaranteed to fall out of support, the absolute ability to recognize that is crucial.
gymbeaux 4mo ago
Those numbers are abysmal. Should we really be using LLMs to write our code? I have a theory- LLMs can spit out code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time. An enterprise app developed entirely with LLM-happy devs might end up virtually unmaintainable. I’m not sure how to explain it, but the more I see LLM-written code the more I feel it’s bad code doing a good job of masquerading as good code. I think this take will become less-hot in the next year or two when we see enterprise greenfield projects that were created entirely with LLM “assistance” go to prod. I think we’ll find that the code is difficult for humans to read, understand, debug, and extend- and I think the larger the codebase the harder it will be for LLMs to maintain. More opportunity for hallucination, larger context windows needed, more tokens bought and spent for smaller and smaller code changes. I think the more code an LLM writes for an app, the worse that codebase becomes.
- realusername 4mo ago
  > code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time They clearly are only assistants for the moment, you can use them to do work ... but only if you could do the said work yourself alone in the first place.
  - ben_w 4mo ago
    I would say "only if you can review said work yourself alone", rather than "do". I'm an experienced developer, but I don't count myself as a web dev or a python dev; I can review the web and python stuff I get out of the AI (sometimes I need to ask the AI follow-up questions so I can find official documentation for what it did), but I can't write it.
    - realusername 4mo ago
      I think you could eventually do it then, it would just take you longer.
      - 4mo ago
andix 4mo ago
I guess you can test that on hypotheticals. Ask about things after the knowledge cut off that never happened. Or ask things that are genuinely unsolvable.
ComputerGuru 4mo ago
I’m not disagreeing with you but at the same time, models don’t “know” anything in that binary sense. I’m not trying to get in the woods here, I genuinely mean that what you pass off as a simple explanation is actually incredibly nuanced. A fact appeared once in training data , a fact never appeared in the training data, a fact appeared ten times, a fact appeared a thousand times. Which does the model know? Facts aren’t stored as-is, they’re all broken down into their components and compressed in the weights. “Similar” facts that didn’t appear an overwhelming number of times get bundled together and eventually conflated. But then what is a similar fact? Which facts were entirely ablated vs which were bundled together with others effectively poisoning the pool but also giving it inference strength? The model doesn’t know anything and can never know what it knows or doesn’t know.
- unshavedyak 4mo ago
  I often wonder how humans "know" things. I suspect (ignorant armchair) we have some ability to signal strength of those facts, via repetition. Without this layer of introspection i imagine LLMs can never avoid hallucination. It obviously breaks down with humans too, given we so easily hallucinate and confuse things we "know". However i still suspect we're more reliable at probing information we've experienced vs not. Even if the case of poisoned knowledge, eg a crime scene accidentally implying information to a witness that the witness doesn't actually know, we still "know" that poisoned information via incorrect inference. Ie we "experienced" it. Wonder what architecture would allow for this style of information/weight probing for an LLM.
  - coldtea 4mo ago
    >I suspect (ignorant armchair) we have some ability to signal strength of those facts, via repetition. Without this layer of introspection Isn't that precisely what the LLM training does? It signals strength of those facts, via repetition.
    - unshavedyak 4mo ago
      Yea but i don't think the LLM can inspect that. Eg it's similar to a code function , the code executes but the code doesn't have the ability to inquire about the code itself. In that silly example/thought, the LLM would effectively need the ability to query the strength of a fact/spatial region/etc. Right now i believe the LLM is more just the output of those weights. It has no way to inspect the strength of the signal. Eg it doesn't know if blue in "The sky is <blue>" is a strong or weak signal, it just predicted that next token. If we could somehow encode strengths along with every token, eg "The sky is <blue:1.0>" or something we could perhaps give it a sense of [un]certainty. Though i imagine it would look differently than that since we'd want to encode this information in some sort of multi-dimensional space, rather than purely by token - since tokens aren't that valuable. Eg the "knowledge" in an LLM goes beyond tokens, and so too should signal strength. I'm of course speculating on all of this and i have no clue on anything.

in-silico 4mo ago

Additionally, maybe it's easier for a model to realize that it doesn't know the answer when the question is easier. If Opus gets all but the hardest questions right, it might have a higher hallucination rate because the questions it gets wrong are the questions where verification or hallucination detection are the most difficult

grayhatter 4mo ago

> Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. Do you have a cite for this? If a human makes up some bullshit lie, I wouldn't accuse them of making it up only if they actually knew the correct answer. If you don't know, the only correct answer is I don't know. Any other answer is made up bullshit. Why is it only a hallucination if and only if the LLM contains the answer? If you make something up it's still wrong. It shouldn't matter if you could give the correct answer. You didn't, and instead invented some bullshit instead? Follow up question, how can I apply this rule set to the next test I have to take? I'd love to be able to use "I didn't know" as the excuse for why I made something up. edit: > and it's not totally clear that this is the main metric that's worth tracking. I don't know, the rate at which some model is willing to make up something feels useful. If the argument I see repeated on HN so much is that it's impossible to completely get rid of hallucinations; being able to choose a model that's less likely to invent some lie seems like a positive trait, no? Either way, I'm happy to agree that a restrictive definition, where a lie doesn't count as a hallucination iff the model doesn't know the answer feels strictly, infinitely less useful than an exact error rate. What percentage of emitted tokens are misleading would be useful for me. Anyone know any group that's attempted to quantify the global error rate?

sgc 4mo ago
Since models just output the the most probable tokens and you can never accuse them of doing anything other than making it all up, I would like to see these tests run with a prompt that attempts to mitigate hallucination and finishes with something like: "Telling me that you don't have the relevant information or that the task is impossible is extremely useful to me and a valid answer", and see how much that changes the scoring - as well as the usefulness of the answers. There are so many skills like context7 that can be tweaked to improve these results as well. In other words, you shouldn't choose the model that hallucinates the least without detailed prompting, since a well-crafted agents.md clause should go a long way to improving output, and almost certainly the top scoring order will be different. To the point that I don't find this type of raw comparison useful beyond maybe 'make sure you test that one with more explicit prompts'.
- grayhatter 4mo ago
  > In other words, you shouldn't choose the model that hallucinates the least without detailed prompting You're prompting it wrong is quickly becoming the new, you're holding it wrong. It's wild how willing software engineers are to blame the user when the actual problem is their own defective design. Ideally we all, as an industry, will stop accepting this as reasonable excuse for the demonstrated incompetence
  - ordersofmag 4mo ago
    It's not that you're prompting it wrong. It's that you're judging the output against a standard (human intelligence) that just isn't relevant--no matter how much we want it to be and no matter how much the fluency of the output tricks us into thinking there's a human-like mind behind it. Now granted, if the boat salesmen were pushing hard on the idea that the boat would fly and even put little wings on the side and I bought the boat I might get really angry when I found out that it didn't fly. And I might angrily storm into the salesroom yelling about how the design is defective. But if someone pointed out 'hey, it's a boat perhaps you should stick to sailing around in it and stop getting your undies in a bundle about it not flying' the correct response is probably to take a closer look, ignore the salesmen, and cruise around the lake. LLM's are quite handy at some things and have some weird limits. Learn the limits, enjoy your time at sea.

reinitctxoffset 4mo ago

Hallucination should be called "failure to ground". Something about the cost model of US near frontier has the cattle prod out whenever a model is uncertain but thrashes on whether to search. Search flinch is roughly all hallucination. I don't even wait for the model's turn, if there's a man page or Hoogle hit, stuff the last prefix cache cut point. You come out ahead.

sudosysgen 4mo ago

This is missing a common failure mode, which is information past the knowledge cutoff. If you need info past that time they'll fail no matter how big or small the model is, so the hallucination rate can matter independently of the knowledge base. If all use-cases had a uniform risk of falling out of support, this would be a valid argument, but since it's often the case that a datapoint is guaranteed to fall out of support, the absolute ability to recognize that is crucial.

gymbeaux 4mo ago

Those numbers are abysmal. Should we really be using LLMs to write our code? I have a theory- LLMs can spit out code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time. An enterprise app developed entirely with LLM-happy devs might end up virtually unmaintainable. I’m not sure how to explain it, but the more I see LLM-written code the more I feel it’s bad code doing a good job of masquerading as good code. I think this take will become less-hot in the next year or two when we see enterprise greenfield projects that were created entirely with LLM “assistance” go to prod. I think we’ll find that the code is difficult for humans to read, understand, debug, and extend- and I think the larger the codebase the harder it will be for LLMs to maintain. More opportunity for hallucination, larger context windows needed, more tokens bought and spent for smaller and smaller code changes. I think the more code an LLM writes for an app, the worse that codebase becomes.

realusername 4mo ago
> code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time They clearly are only assistants for the moment, you can use them to do work ... but only if you could do the said work yourself alone in the first place.
- ben_w 4mo ago
  I would say "only if you can review said work yourself alone", rather than "do". I'm an experienced developer, but I don't count myself as a web dev or a python dev; I can review the web and python stuff I get out of the AI (sometimes I need to ask the AI follow-up questions so I can find official documentation for what it did), but I can't write it.
  - realusername 4mo ago
    I think you could eventually do it then, it would just take you longer.
    - 4mo ago

andix 4mo ago

I guess you can test that on hypotheticals. Ask about things after the knowledge cut off that never happened. Or ask things that are genuinely unsolvable.

ComputerGuru 4mo ago

I’m not disagreeing with you but at the same time, models don’t “know” anything in that binary sense. I’m not trying to get in the woods here, I genuinely mean that what you pass off as a simple explanation is actually incredibly nuanced. A fact appeared once in training data , a fact never appeared in the training data, a fact appeared ten times, a fact appeared a thousand times. Which does the model know? Facts aren’t stored as-is, they’re all broken down into their components and compressed in the weights. “Similar” facts that didn’t appear an overwhelming number of times get bundled together and eventually conflated. But then what is a similar fact? Which facts were entirely ablated vs which were bundled together with others effectively poisoning the pool but also giving it inference strength? The model doesn’t know anything and can never know what it knows or doesn’t know.

unshavedyak 4mo ago
I often wonder how humans "know" things. I suspect (ignorant armchair) we have some ability to signal strength of those facts, via repetition. Without this layer of introspection i imagine LLMs can never avoid hallucination. It obviously breaks down with humans too, given we so easily hallucinate and confuse things we "know". However i still suspect we're more reliable at probing information we've experienced vs not. Even if the case of poisoned knowledge, eg a crime scene accidentally implying information to a witness that the witness doesn't actually know, we still "know" that poisoned information via incorrect inference. Ie we "experienced" it. Wonder what architecture would allow for this style of information/weight probing for an LLM.
- coldtea 4mo ago
  >I suspect (ignorant armchair) we have some ability to signal strength of those facts, via repetition. Without this layer of introspection Isn't that precisely what the LLM training does? It signals strength of those facts, via repetition.
  - unshavedyak 4mo ago
    Yea but i don't think the LLM can inspect that. Eg it's similar to a code function , the code executes but the code doesn't have the ability to inquire about the code itself. In that silly example/thought, the LLM would effectively need the ability to query the strength of a fact/spatial region/etc. Right now i believe the LLM is more just the output of those weights. It has no way to inspect the strength of the signal. Eg it doesn't know if blue in "The sky is <blue>" is a strong or weak signal, it just predicted that next token. If we could somehow encode strengths along with every token, eg "The sky is <blue:1.0>" or something we could perhaps give it a sense of [un]certainty. Though i imagine it would look differently than that since we'd want to encode this information in some sort of multi-dimensional space, rather than purely by token - since tokens aren't that valuable. Eg the "knowledge" in an LLM goes beyond tokens, and so too should signal strength. I'm of course speculating on all of this and i have no clue on anything.