4 ms·

The non-hallucination rate in AA-omniscience is SOTA, better than Opus 4.7, Gemini 3.1 Pro and GPT5.5! Congrats to the team

throawayonthe 5mo ago
referencing this: https://artificialanalysis.ai/evaluations/omniscience?models=gemini-3-1-pro-preview%2Cclaude-opus-4-7%2Cgemini-3-5-flash%2Cgpt-5-5%2Cgrok-4-3%2Cqwen3-7-max%2Cclaude-sonnet-4-6-adaptive%2Cqwen3-6-max%2Ckimi-k2-6%2Cgpt-5-4%2Cmuse-spark%2Cmimo-v2-5-pro%2Cglm-5-1%2Cminimax-m2-7%2Cclaude-4-5-haiku-reasoning%2Cdeepseek-v4-pro%2Cllama-3-1-instruct-405b%2Cgpt-5-4-mini%2Cdeepseek-v3-2-reasoning%2Cdeepseek-v4-flash%2Cqwen3-5-397b-a17b%2Ck2-think-v2%2Cmistral-medium-3-5%2Cnvidia-nemotron-3-super-120b-a12b%2Cgemma-4-31b%2Cnova-2-0-pro-reasoning-medium%2Cgpt-oss-120b%2Csolar-pro-3%2Cgpt-oss-20b#omniscience-hallucination-rate-tabs https://artificialanalysis.ai/evaluations/omniscience?models... (had to add it to the chart, wasn't displayed by default. is it the lowest rate in the datasetor no?)
- jampekka 5mo ago
  This counts only incorrect answers though. A model can get 0% hallucination rate just by refusing to answer all questions.
  - ffsm8 5mo ago
    Isn't that precisely the reason why we introduced the term hallucination? Because llms have historically always made up bullshit of they cannot answer directly... If they now nailed this to maybe the model not respond instead of responding incorrectly, then a lot of previously unusable usecases would become feasible. So I feel like that's exactly the right metric and the way to track it wrt hallucinations.
    - doublescoop 5mo ago
      I had a buddy in high school that was notorious for doing the same thing. (He's now a senior director at a Big 4 consultancy. :) )
      - rrgok 5mo ago
        Do you mind expanding a little more?
        alfiedotwtf 5mo ago
        They had a buddy who used to lie a lot when they were younger… now they get paid for it
    - akoboldfrying 5mo ago
      The point is that it's not a useful metric on its own. For example, redirecting from /dev/null also achieves a zero hallucination rate. We want the hallucination rate to decrease while the overall answer rate of queries remains sufficiently high. For more specifics, look into ROC and AUC.
  - speed_spread 5mo ago
    Yes. A model that can answer "I don't know" would be much more trustable than the current used car salesman we have now.
    - jorvi 5mo ago
      Its very annoying this has been in the capability of models since the very beginning. It could check how probable its token values are and if those fall below a certain threshold either say "I don't know", or output the most probable (well, more like least improbable) tokens but give a very clear, very strong warning that it is a shot in the dark and likely to contain hallucinations. But no, Google and OpenAI would rather always have an answer ready and tell you to mix glue into your pizza toppings :)
      - nomel 5mo ago
        Yeah, I never understood why the top n statistics weren't included in the chat interfaces, to color the text!
      - tokenscoper 5mo ago
        I don't have much to add other than this observation that we seem to have moved away from eating one small rock per day for nutritional value, and adding gasoline in spaghetti. The glue on pizza reference brought back memories :)
      - miki123211 5mo ago
        It can't, because top n isn't always reliable. Hallucination detection is an open problem. If it were that simple, people would indeed "just" do it. Basically the problem is that LLMs aren't trained on things they don't know; an alternative way of saying this is that they're not trained on things they're not trained on, which is obviously true. When you RL a model and it answers incorrectly, you don't teach it to answer "I don't know", you teach it to answer correctly instead. This makes it very hard for it to realize when it doesn't know things.
        chengyongru 5mo ago
  - jug 5mo ago
    I think that's what the Omniscience Index is for: https://artificialanalysis.ai/evaluations/omniscience#aa-omniscience-index-results https://artificialanalysis.ai/evaluations/omniscience#aa-omn... It rewards correct answers and penalizes hallucinations, and finally no reward for refusing to answer. It's interesting just how poorly some popular Chinese models fare in this regard, like GLM 5.1 or DeepSeek 4 Pro. Gemini 3.x has truly remarkable knowledge given how it leads in this benchmark despite being (quite a bit) more prone to hallucinate than Claude Opus.
  - aicantdeny 5mo ago
    > by refusing to answer all questions. Cool, precisely the thing other AI is too stupid to do when they don't have the necessary knowledge.
  - Balinares 5mo ago
    Yes, that's in fact precisely the desired behavior when a model doesn't know the answer.
sheepscreek 5mo ago
Truly incredible! Very impressed by their progress. I wonder how much of their own chips did they use for training.
baq 5mo ago
wonder at which level there's a capability state transition? 5%? 1%?
gslepak 5mo ago
> The non-hallucination rate in AA-omniscience is SOTA Note that a perfect "non-hallucination rate" is rather meaningless as such tests can contain human hallucinations. It means the model aligns with the possibly-true, possibly-false beliefs of the group that made the test.
- rlt 5mo ago
  Well, yes, garbage in garbage out. That's a given and not what's meant by "hallucination" in this context.
  - tantaman 5mo ago
    the observation goes beyond garbage in garbage out. Mainly that we're always operating from some prior and limited understanding. That what may look like a hallucination could be closer to the truth than our current frameworks of understanding allow us to admit. The hermeneutic circle.
    - Jacques2Marais 5mo ago
      Interesting. I wonder if current LLMs can break out of human limitations and understand the world more correctly.
    - root_axis 5mo ago
      A properly designed benchmark won't use tests that leave room for ambiguous interpretation.
- jcheng 5mo ago
  Here are some examples of the questions in the benchmark. If these are representative, they seem pretty cut and dry. https://artificialanalysis.ai/evaluations/omniscience#example-tasks https://artificialanalysis.ai/evaluations/omniscience#exampl...
- areweai 5mo ago
  Was there something about this specific model and submission that made you feel compelled to write this self-evident observation? Or would you describe your methodology as more like picking a random sentence fragment as an input value then generating completions from your existing corpus without any post-input "learning" process related to the rest of the source material?
girvo 5mo ago
The big question for me having used a lot of these SOTA chinese models is: what is its token efficiency like? Running Step 3.5 Flash locally for example, it's an amazingly capable model all things considered, but it's token efficiency is so bad that it gets out performed by most others wall-clock time (even with my MTP-support for it hacked in to llama.cpp: despite being trained on three heads, MTP 2 is the sweet spot, and only gets it from 20tk/s to 30tk/s on my Spark) The DeepSeek models and Qwen 3.5 Plus are also good examples of this: compared to Opus, and especially GPT 5.5 they use many more tokens to get to the same answers. I'm really hoping that Qwen 3.7 is better in this regard, can't wait to try it out (ps. running DeepSeek v4 Flash on my Spark is absolutely wild, thanks antirez if you see this haha)
- nl 5mo ago
  Yes it's a big thing that people are slowly becoming more aware of. Nvidia models are even worse than Qwen! https://sql-benchmark.nicklothian.com/#token-efficiency-and-speed https://sql-benchmark.nicklothian.com/#token-efficiency-and-... (mouse over the cells for token counts and click for traces) Gemma 4 is good for this, as AA notes: > Gemma 4 31B is notably token efficient, using 39M output tokens to run the Intelligence Index vs 98M for Qwen3.5 27B (Reasoning). This is ~2.5x fewer output tokens for a model scoring 3 points lower. For context, the other models at the 42-point intelligence level also use significantly more tokens: MiniMax-M2.5 (56M), DeepSeek V3.2 (Reasoning, 61M), and GLM-4.7 (Reasoning, 167M) https://artificialanalysis.ai/articles/gemma-4-everything-you-need-to-know https://artificialanalysis.ai/articles/gemma-4-everything-yo...
  - girvo 5mo ago
    Right and all of my own evals back this up for Gemma 4... ...except its notably worse at coding in an agent context even with a harness setup to do exactly what Google says it should do (wrt. to sending summarised thinking back and so on) So despite it being far better token efficiency wise, it's just worse for what I need to use it for compared to DSv4 Flash or Qwen 3.6 27B Such a shame, too.

throawayonthe 5mo ago

referencing this: https://artificialanalysis.ai/evaluations/omniscience?models=gemini-3-1-pro-preview%2Cclaude-opus-4-7%2Cgemini-3-5-flash%2Cgpt-5-5%2Cgrok-4-3%2Cqwen3-7-max%2Cclaude-sonnet-4-6-adaptive%2Cqwen3-6-max%2Ckimi-k2-6%2Cgpt-5-4%2Cmuse-spark%2Cmimo-v2-5-pro%2Cglm-5-1%2Cminimax-m2-7%2Cclaude-4-5-haiku-reasoning%2Cdeepseek-v4-pro%2Cllama-3-1-instruct-405b%2Cgpt-5-4-mini%2Cdeepseek-v3-2-reasoning%2Cdeepseek-v4-flash%2Cqwen3-5-397b-a17b%2Ck2-think-v2%2Cmistral-medium-3-5%2Cnvidia-nemotron-3-super-120b-a12b%2Cgemma-4-31b%2Cnova-2-0-pro-reasoning-medium%2Cgpt-oss-120b%2Csolar-pro-3%2Cgpt-oss-20b#omniscience-hallucination-rate-tabs https://artificialanalysis.ai/evaluations/omniscience?models... (had to add it to the chart, wasn't displayed by default. is it the lowest rate in the datasetor no?)

jampekka 5mo ago
This counts only incorrect answers though. A model can get 0% hallucination rate just by refusing to answer all questions.
- ffsm8 5mo ago
  Isn't that precisely the reason why we introduced the term hallucination? Because llms have historically always made up bullshit of they cannot answer directly... If they now nailed this to maybe the model not respond instead of responding incorrectly, then a lot of previously unusable usecases would become feasible. So I feel like that's exactly the right metric and the way to track it wrt hallucinations.
  - doublescoop 5mo ago
    I had a buddy in high school that was notorious for doing the same thing. (He's now a senior director at a Big 4 consultancy. :) )
    - rrgok 5mo ago
      Do you mind expanding a little more?
      - alfiedotwtf 5mo ago
        They had a buddy who used to lie a lot when they were younger… now they get paid for it
  - akoboldfrying 5mo ago
    The point is that it's not a useful metric on its own. For example, redirecting from /dev/null also achieves a zero hallucination rate. We want the hallucination rate to decrease while the overall answer rate of queries remains sufficiently high. For more specifics, look into ROC and AUC.
- speed_spread 5mo ago
  Yes. A model that can answer "I don't know" would be much more trustable than the current used car salesman we have now.
  - jorvi 5mo ago
    Its very annoying this has been in the capability of models since the very beginning. It could check how probable its token values are and if those fall below a certain threshold either say "I don't know", or output the most probable (well, more like least improbable) tokens but give a very clear, very strong warning that it is a shot in the dark and likely to contain hallucinations. But no, Google and OpenAI would rather always have an answer ready and tell you to mix glue into your pizza toppings :)
    - nomel 5mo ago
      Yeah, I never understood why the top n statistics weren't included in the chat interfaces, to color the text!
    - tokenscoper 5mo ago
      I don't have much to add other than this observation that we seem to have moved away from eating one small rock per day for nutritional value, and adding gasoline in spaghetti. The glue on pizza reference brought back memories :)
    - miki123211 5mo ago
      It can't, because top n isn't always reliable. Hallucination detection is an open problem. If it were that simple, people would indeed "just" do it. Basically the problem is that LLMs aren't trained on things they don't know; an alternative way of saying this is that they're not trained on things they're not trained on, which is obviously true. When you RL a model and it answers incorrectly, you don't teach it to answer "I don't know", you teach it to answer correctly instead. This makes it very hard for it to realize when it doesn't know things.
      - chengyongru 5mo ago
- jug 5mo ago
  I think that's what the Omniscience Index is for: https://artificialanalysis.ai/evaluations/omniscience#aa-omniscience-index-results https://artificialanalysis.ai/evaluations/omniscience#aa-omn... It rewards correct answers and penalizes hallucinations, and finally no reward for refusing to answer. It's interesting just how poorly some popular Chinese models fare in this regard, like GLM 5.1 or DeepSeek 4 Pro. Gemini 3.x has truly remarkable knowledge given how it leads in this benchmark despite being (quite a bit) more prone to hallucinate than Claude Opus.
- aicantdeny 5mo ago
  > by refusing to answer all questions. Cool, precisely the thing other AI is too stupid to do when they don't have the necessary knowledge.
- Balinares 5mo ago
  Yes, that's in fact precisely the desired behavior when a model doesn't know the answer.

sheepscreek 5mo ago

Truly incredible! Very impressed by their progress. I wonder how much of their own chips did they use for training.

baq 5mo ago

wonder at which level there's a capability state transition? 5%? 1%?

gslepak 5mo ago

> The non-hallucination rate in AA-omniscience is SOTA Note that a perfect "non-hallucination rate" is rather meaningless as such tests can contain human hallucinations. It means the model aligns with the possibly-true, possibly-false beliefs of the group that made the test.

rlt 5mo ago
Well, yes, garbage in garbage out. That's a given and not what's meant by "hallucination" in this context.
- tantaman 5mo ago
  the observation goes beyond garbage in garbage out. Mainly that we're always operating from some prior and limited understanding. That what may look like a hallucination could be closer to the truth than our current frameworks of understanding allow us to admit. The hermeneutic circle.
  - Jacques2Marais 5mo ago
    Interesting. I wonder if current LLMs can break out of human limitations and understand the world more correctly.
  - root_axis 5mo ago
    A properly designed benchmark won't use tests that leave room for ambiguous interpretation.
jcheng 5mo ago
Here are some examples of the questions in the benchmark. If these are representative, they seem pretty cut and dry. https://artificialanalysis.ai/evaluations/omniscience#example-tasks https://artificialanalysis.ai/evaluations/omniscience#exampl...
areweai 5mo ago
Was there something about this specific model and submission that made you feel compelled to write this self-evident observation? Or would you describe your methodology as more like picking a random sentence fragment as an input value then generating completions from your existing corpus without any post-input "learning" process related to the rest of the source material?

girvo 5mo ago

The big question for me having used a lot of these SOTA chinese models is: what is its token efficiency like? Running Step 3.5 Flash locally for example, it's an amazingly capable model all things considered, but it's token efficiency is so bad that it gets out performed by most others wall-clock time (even with my MTP-support for it hacked in to llama.cpp: despite being trained on three heads, MTP 2 is the sweet spot, and only gets it from 20tk/s to 30tk/s on my Spark) The DeepSeek models and Qwen 3.5 Plus are also good examples of this: compared to Opus, and especially GPT 5.5 they use many more tokens to get to the same answers. I'm really hoping that Qwen 3.7 is better in this regard, can't wait to try it out (ps. running DeepSeek v4 Flash on my Spark is absolutely wild, thanks antirez if you see this haha)

nl 5mo ago
Yes it's a big thing that people are slowly becoming more aware of. Nvidia models are even worse than Qwen! https://sql-benchmark.nicklothian.com/#token-efficiency-and-speed https://sql-benchmark.nicklothian.com/#token-efficiency-and-... (mouse over the cells for token counts and click for traces) Gemma 4 is good for this, as AA notes: > Gemma 4 31B is notably token efficient, using 39M output tokens to run the Intelligence Index vs 98M for Qwen3.5 27B (Reasoning). This is ~2.5x fewer output tokens for a model scoring 3 points lower. For context, the other models at the 42-point intelligence level also use significantly more tokens: MiniMax-M2.5 (56M), DeepSeek V3.2 (Reasoning, 61M), and GLM-4.7 (Reasoning, 167M) https://artificialanalysis.ai/articles/gemma-4-everything-you-need-to-know https://artificialanalysis.ai/articles/gemma-4-everything-yo...
- girvo 5mo ago
  Right and all of my own evals back this up for Gemma 4... ...except its notably worse at coding in an agent context even with a harness setup to do exactly what Google says it should do (wrt. to sending summarised thinking back and so on) So despite it being far better token efficiency wise, it's just worse for what I need to use it for compared to DSv4 Flash or Qwen 3.6 27B Such a shame, too.