4 ms·

Very interesting that one of the components is "AA-Omniscience Index" AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination.

by firasd 3mo ago

Very interesting that one of the components is "AA-Omniscience Index" AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. This seems to be a good proxy for param size/density and the ranking breaks down as such: Claude Fable 5 (with fallback), Gemini 3.1 Pro Preview, Claude Opus 5 (Max), Grok 4.6 (high), Gemini 3.6 Flash, GPT 5.6 Sol (Max) I've thought for a while that Gemini 3.x has 'big model smell'

mchusma 3mo ago
Gemini 3.1 pro is really good for knowledge tasks. Google has done well there. And image analysis with Gemini flash 3.6 is solid. It’s just anything coding or agentic they fall short.
- andriy_koval 3mo ago
  they likely tune their models for areas where they have their money: search, ads, youtube, etc.
  - alex43578 3mo ago
    I wonder if that'll be a mistake as LLMs are used for internal LLM R&D. Either Google will not take this approach, use a 3rd party model (weird, data leak risk?), or use a non-public internal model (big sunk dev cost with no recoup by trickling it to public).
    - andriy_koval 3mo ago
      there were news that google uses claude internally, and also other news that Apple has its own claude tuned on internal data, maybe google has the same..
- victor106 3mo ago
  > knowledge tasks Like what?
chronogram 3mo ago
I'm not so sure. Especially with 3.6 Flash being 24 to Sol's 22. The 3.x Flash models were thought to fit on a single TPU 8i and it's the odd one out in that list at a whopping 234.7t/s compared to Sol's 64.4t/s and Opus's 56.3t/s. Even 3.1 Pro is a much higher 113.9t/s than the others. I think AA-Omniscience Accuracy follows your expectations better. An ultra size Fable at 61%, followed by large frontier models like Sol, 5.5 and Opus. With Flash being up there. I assume because Gemini is more focused on general knowledge to operational cost in particular, rather than getting the highest scores in coding benchmarks. If you go to Domain Score (Normalized) you'll see that the Gemini models are only less competitive in Software. And that's where Sol goes from 6 in Health to 71 in Software.
mdgld 3mo ago
I think grounding has as much if not more go do with it. Google does a great job w/ grounding for obvious reasons
- firasd 3mo ago
  Grounding as in web search? I think this Omniscience benchmark would not include access to a search tool cause otherwise it becomes kinda meaningless
krzyk 3mo ago
If it doesn't give penalty for refusal it is not that useful.

by firasd 3mo ago

mchusma 3mo ago

Gemini 3.1 pro is really good for knowledge tasks. Google has done well there. And image analysis with Gemini flash 3.6 is solid. It’s just anything coding or agentic they fall short.

andriy_koval 3mo ago
they likely tune their models for areas where they have their money: search, ads, youtube, etc.
- alex43578 3mo ago
  I wonder if that'll be a mistake as LLMs are used for internal LLM R&D. Either Google will not take this approach, use a 3rd party model (weird, data leak risk?), or use a non-public internal model (big sunk dev cost with no recoup by trickling it to public).
  - andriy_koval 3mo ago
    there were news that google uses claude internally, and also other news that Apple has its own claude tuned on internal data, maybe google has the same..
victor106 3mo ago
> knowledge tasks Like what?

chronogram 3mo ago

I'm not so sure. Especially with 3.6 Flash being 24 to Sol's 22. The 3.x Flash models were thought to fit on a single TPU 8i and it's the odd one out in that list at a whopping 234.7t/s compared to Sol's 64.4t/s and Opus's 56.3t/s. Even 3.1 Pro is a much higher 113.9t/s than the others. I think AA-Omniscience Accuracy follows your expectations better. An ultra size Fable at 61%, followed by large frontier models like Sol, 5.5 and Opus. With Flash being up there. I assume because Gemini is more focused on general knowledge to operational cost in particular, rather than getting the highest scores in coding benchmarks. If you go to Domain Score (Normalized) you'll see that the Gemini models are only less competitive in Software. And that's where Sol goes from 6 in Health to 71 in Software.

mdgld 3mo ago

I think grounding has as much if not more go do with it. Google does a great job w/ grounding for obvious reasons

firasd 3mo ago
Grounding as in web search? I think this Omniscience benchmark would not include access to a search tool cause otherwise it becomes kinda meaningless

krzyk 3mo ago

If it doesn't give penalty for refusal it is not that useful.