3 ms·

I already felt that gemini 3 proved what is possible if you train a model for efficiency. If I had to guess the pro and flash variants are 5x to 10x smaller tha

I already felt that gemini 3 proved what is possible if you train a model for efficiency. If I had to guess the pro and flash variants are 5x to 10x smaller than opus and gpt-5 class models. They produce drastically lower amount of tokens to solve a problem, but they haven't seem to have put enough effort into refinining their reasoning and execution as they produce broken toolcalls and generally struggle with 'agentic' tasks, but for raw problem solving without tools or search they match opus and gpt while presumably being a fraction of the size. I feel like google will surprise everyone with a model that will be an entire generation beyond SOTA at some point in time once they go from prototyping to making a model that's not a preview model anymore. All models up till now feel like they're just prototypes that were pushed to GA just so they have something to show to investors and to integrate into their suite as a proof of concept.

ALLTaken 6mo ago
[flagged]
- _boffin_ 6mo ago
  Is your friend on the JAX team?
- neonstatic 6mo ago
  I'm really struggling with terrible bloating today, but I deemed it too dangerous to release.
  - tclancy 6mo ago
    Thank you for your sacrifice. Could you speak to my dog please? You may wish to yell from a distance, actually.
onlyrealcuzzo 6mo ago
> They produce drastically lower amount of tokens to solve a problem, but they haven't seem to have put enough effort into refinining their reasoning and execution as they produce broken toolcalls and generally struggle with 'agentic' tasks, but for raw problem solving without tools or search they match opus and gpt while presumably being a fraction of the size. Agreed, Gemini-cli is terrible compared to CC and even Codex. But Google is clearly prioritizing to have the best AI to augment and/or replace traditional search. That's their bread and butter. They'll be in a far better place to monetize that than anyone else. They've got a 1B+ user lead on anyone - and even adding in all LLMs together, they still probably have more query volume than everyone else put together. I hope they start prioritizing Gemini-cli, as I think they'd force a lot more competition into the space.
- asah 6mo ago
  also, for incorporating into gsuite, youtube, maps, gcp and their other winning apps and behind-the-scenes infra...
- JeremyNT 6mo ago
  > Agreed, Gemini-cli is terrible compared to CC and even Codex. Using it with opencode I don't find the actual model to cause worse results with tool calling versus Opus/GPT. This could be a harness problem more than a model problem? I do prefer the overall results with GPT 5.4, which seems to catch more bugs in reviews that Gemini misses and produce cleaner code overall. (And no, I can't quantify any of that, just "vibes" based)
- Iulioh 6mo ago
  Not only that, google has an advange because they don't need to always generate a response. When a lot of people ask the same thing they can just index the questions, like a results on the search engine and recalculate it only so often,
- rjh29 6mo ago
  I wonder what I am missing, because I can use gemini-cli with English descriptions of features or entire projects and it just cranks out the code. Built a bunch of stuff with it. Can't think of anything it's currently lacking.
  - CraigJPerry 6mo ago
    >> Can't think of anything it's currently lacking. Speed? The pro models are slow for me The model 3.1 pro model is good and i don't recognise the GP's complaint of broken tool calls but i'm only using via gemini cli harness, sounds like they might be hosting their own agentic loop?
  - xnx 6mo ago
    Same. I've built dozens of small tools and scripts and never felt the need to try something else.
- 6mo ago
UncleOxidant 6mo ago
IIRC when Gemini 3 Pro came out it was considered to be just about on par with whatever version of Claude was out then (4?). Now Gemini 3 is looking long in the tooth. Considering how many Chinese models have been released since then, and at least 2 or 3 versions of Claude, it's starting to look like Google is kind of sitting still here. Maybe you're right and they'll surprise us soon with a large step improvement over what they currently have. Note: I do realize that there's been a Gemini 3.1 release, but it didn't seem like a noticeable change from 3.
- cmrdporcupine 6mo ago
  As other people are saying here: the Gemini models are mostly terrible at tool use and long context management. And maybe not quite as good with finicky "detail" parts of coding generally. Where they excel is just total holistic _knowledge_ about the world. I don't like "talking" to it, because I kind of hate its tone, but I find Gemini generally extremely useful for research and analysis tasks and looking up information.
  - CuriouslyC 6mo ago
    Gemini had the best long context support for the longest time, and even now at >400k tokens it's still got the best long context recall. Gemini is just not trained for autonomy/tool use/agentic behavior to the same degree as the other frontier models. Goog seems to emphasize video/images/scientific+world knowledge.
    - cmrdporcupine 6mo ago
      My experience is it advertises large context and then just becomes incoherent and confused as it climbs to fill that context. e.g. it sucks at general tool use but sucks even more at it after a chunk of time in a session. One frustrating situation is to watch it go into a loop trying and failing to edit source files. I often wonder how my old coworkers from Google get by, if this is the the agentic coding they have available to them for working on projects on Google3. But I suspect the models they work with have been fine tuned on Google's custom tooling and perform better?
orbital-decay 6mo ago
Their "preview" naming is pretty arbitrary. It's just their way to avoid making any availability or persistence promises, let alone guarantees. It's also a PR tactic to mask any failures by pretending it's beta quality.
mrcwinn 6mo ago
Interesting mix of words: "I felt" -> "proved" -> "guess". One of those is not like the others!
- himata4113 6mo ago
  I guess I felt pretty uncertain that day which proved that a lack of sleep is bad for your mental cognition.
big-chungus4 6mo ago
Am I tripping or is this an AI reply? Like it barely has anything to do with the article other than both are related to AI
- Jensson 6mo ago
  An AI reply would be more relevant to the headline / article, humans often write something tangential since we have more going on in our head and not just the context at hand while AI can't ignore context.
- himata4113 6mo ago
  Google uses these chips to create gemini, I simply used this as an excuse to rant and predict the future.
deaux 6mo ago
> If I had to guess the pro and flash variants are 5x to 10x smaller than opus and gpt-5 class models. I really doubt it, especially Pro. If anything I wouldn't be surprised if their hardware lets them run bigger models more cheaply and quickly than the others. Pro is probably smaller than GPT 5.4 and Opus 4.6 (looks like 4.7 decreased in size), but 5x seems way too much. IMO Gemini 3 Pro is the most "intelligent" in an all-round human way. Especially in the humanities. It's highly knowledgeable and undeniably the number one model at producing natural text in a large number of (human!) languages. The difference becomes especially large for more niche languages. That does not suggest a smaller model, more the opposite. The top 4 models at multilinguality are all Google : 1. 3 Pro 2. 3 Flash 3. 2.5 Pro 4. 2.5 Flash. Even the biggest OpenAI and Anthropic models can't compete in that dimension. It's definitely weaker at math and much worse at agentic things. Gemini chat as an app is also lightyears behind, it's barely different from ChatGPT at release over 3 yeaes ago. These things make it feel much weaker than it is.
- algoth1 6mo ago
  Aistudio should be their default app
- orbital-decay 6mo ago
  Regarding Anthropic, they used to make best multilingual and generalist models, it's their policy thing, not a capability issue. Claude 3 was best at this, including dead and low-resource languages. Neither modern Claude nor Gemini are remotely close to what Claude 3 was capable of (e.g. zero-shot writing styles). Anthropic basically reversed their "character training" policy and started optimizing their models for code generation at the cost of everything else, starting with Sonnet 3.5. Claude 4 took a huge hit in multilingual ability GPT, on the other hand, was always terrible at languages, except for the short-lived gpt-4.5-preview. All modern models including Gemini have bugs in basic language coherency - random language switching, self-correction attempts resulting in hallucinations etc. I speculate it's a problem with heavy RL with rewards and policies not optimized for creative writing.
  - awongh 6mo ago
robocat 6mo ago
> a model that will be an entire generation beyond SOTA That model would then be SOTA. Tautologically you can't be better than SOTA
- himata4113 6mo ago
  SOTA at that time*
solarkraft 6mo ago
I really wonder what I’m missing with Gemini. It’s a second rate model for me at best. I find it okay (not great) at collecting information and completely useless at agentic tasks. It’s like it’s always drunk. When the Claude credits expire in Antigravity, I’m done for the day. > They produce drastically lower amount of tokens to solve a problem I LOLed at this because I of the constant death loops that don’t even solve the problem at all.
- 8note 6mo ago
  i get much better results with it using a different toolset. give it serena and it mostly works, and is less likely to hit a death loop. i feel like the geminicli app is missing some tools for making sure the session history is actually valid
- himata4113 6mo ago
  Yah it doesn't even make sense how they got through their benchmarks without death loops. Gemini-cli even has a hotfix to break the model of such death-loops. But if you were to ignore this bug/quirk that will be fixed in the next patch release my point still stands.
nl 6mo ago
> the pro and flash variants are 5x to 10x smaller than opus and gpt-5 class models. The rumor is that Gemini Pro is the largest model being served today (or at least was prior to Mythos) Source: some podcast where they were discussing TPU vs Nvidia cluster topologies, and how Google is exploiting their topology to allow this. But I can't remember exactly which podcast, so hopefully someone else will know.

ALLTaken 6mo ago

[flagged]

_boffin_ 6mo ago
Is your friend on the JAX team?
neonstatic 6mo ago
I'm really struggling with terrible bloating today, but I deemed it too dangerous to release.
- tclancy 6mo ago
  Thank you for your sacrifice. Could you speak to my dog please? You may wish to yell from a distance, actually.

onlyrealcuzzo 6mo ago

> They produce drastically lower amount of tokens to solve a problem, but they haven't seem to have put enough effort into refinining their reasoning and execution as they produce broken toolcalls and generally struggle with 'agentic' tasks, but for raw problem solving without tools or search they match opus and gpt while presumably being a fraction of the size. Agreed, Gemini-cli is terrible compared to CC and even Codex. But Google is clearly prioritizing to have the best AI to augment and/or replace traditional search. That's their bread and butter. They'll be in a far better place to monetize that than anyone else. They've got a 1B+ user lead on anyone - and even adding in all LLMs together, they still probably have more query volume than everyone else put together. I hope they start prioritizing Gemini-cli, as I think they'd force a lot more competition into the space.

asah 6mo ago
also, for incorporating into gsuite, youtube, maps, gcp and their other winning apps and behind-the-scenes infra...
JeremyNT 6mo ago
> Agreed, Gemini-cli is terrible compared to CC and even Codex. Using it with opencode I don't find the actual model to cause worse results with tool calling versus Opus/GPT. This could be a harness problem more than a model problem? I do prefer the overall results with GPT 5.4, which seems to catch more bugs in reviews that Gemini misses and produce cleaner code overall. (And no, I can't quantify any of that, just "vibes" based)
Iulioh 6mo ago
Not only that, google has an advange because they don't need to always generate a response. When a lot of people ask the same thing they can just index the questions, like a results on the search engine and recalculate it only so often,
rjh29 6mo ago
I wonder what I am missing, because I can use gemini-cli with English descriptions of features or entire projects and it just cranks out the code. Built a bunch of stuff with it. Can't think of anything it's currently lacking.
- CraigJPerry 6mo ago
  >> Can't think of anything it's currently lacking. Speed? The pro models are slow for me The model 3.1 pro model is good and i don't recognise the GP's complaint of broken tool calls but i'm only using via gemini cli harness, sounds like they might be hosting their own agentic loop?
- xnx 6mo ago
  Same. I've built dozens of small tools and scripts and never felt the need to try something else.
6mo ago

UncleOxidant 6mo ago

IIRC when Gemini 3 Pro came out it was considered to be just about on par with whatever version of Claude was out then (4?). Now Gemini 3 is looking long in the tooth. Considering how many Chinese models have been released since then, and at least 2 or 3 versions of Claude, it's starting to look like Google is kind of sitting still here. Maybe you're right and they'll surprise us soon with a large step improvement over what they currently have. Note: I do realize that there's been a Gemini 3.1 release, but it didn't seem like a noticeable change from 3.

cmrdporcupine 6mo ago
As other people are saying here: the Gemini models are mostly terrible at tool use and long context management. And maybe not quite as good with finicky "detail" parts of coding generally. Where they excel is just total holistic _knowledge_ about the world. I don't like "talking" to it, because I kind of hate its tone, but I find Gemini generally extremely useful for research and analysis tasks and looking up information.
- CuriouslyC 6mo ago
  Gemini had the best long context support for the longest time, and even now at >400k tokens it's still got the best long context recall. Gemini is just not trained for autonomy/tool use/agentic behavior to the same degree as the other frontier models. Goog seems to emphasize video/images/scientific+world knowledge.
  - cmrdporcupine 6mo ago
    My experience is it advertises large context and then just becomes incoherent and confused as it climbs to fill that context. e.g. it sucks at general tool use but sucks even more at it after a chunk of time in a session. One frustrating situation is to watch it go into a loop trying and failing to edit source files. I often wonder how my old coworkers from Google get by, if this is the the agentic coding they have available to them for working on projects on Google3. But I suspect the models they work with have been fine tuned on Google's custom tooling and perform better?

orbital-decay 6mo ago

Their "preview" naming is pretty arbitrary. It's just their way to avoid making any availability or persistence promises, let alone guarantees. It's also a PR tactic to mask any failures by pretending it's beta quality.

mrcwinn 6mo ago

Interesting mix of words: "I felt" -> "proved" -> "guess". One of those is not like the others!

himata4113 6mo ago
I guess I felt pretty uncertain that day which proved that a lack of sleep is bad for your mental cognition.

big-chungus4 6mo ago

Am I tripping or is this an AI reply? Like it barely has anything to do with the article other than both are related to AI

Jensson 6mo ago
An AI reply would be more relevant to the headline / article, humans often write something tangential since we have more going on in our head and not just the context at hand while AI can't ignore context.
himata4113 6mo ago
Google uses these chips to create gemini, I simply used this as an excuse to rant and predict the future.

deaux 6mo ago

> If I had to guess the pro and flash variants are 5x to 10x smaller than opus and gpt-5 class models. I really doubt it, especially Pro. If anything I wouldn't be surprised if their hardware lets them run bigger models more cheaply and quickly than the others. Pro is probably smaller than GPT 5.4 and Opus 4.6 (looks like 4.7 decreased in size), but 5x seems way too much. IMO Gemini 3 Pro is the most "intelligent" in an all-round human way. Especially in the humanities. It's highly knowledgeable and undeniably the number one model at producing natural text in a large number of (human!) languages. The difference becomes especially large for more niche languages. That does not suggest a smaller model, more the opposite. The top 4 models at multilinguality are all Google : 1. 3 Pro 2. 3 Flash 3. 2.5 Pro 4. 2.5 Flash. Even the biggest OpenAI and Anthropic models can't compete in that dimension. It's definitely weaker at math and much worse at agentic things. Gemini chat as an app is also lightyears behind, it's barely different from ChatGPT at release over 3 yeaes ago. These things make it feel much weaker than it is.

algoth1 6mo ago
Aistudio should be their default app
orbital-decay 6mo ago
Regarding Anthropic, they used to make best multilingual and generalist models, it's their policy thing, not a capability issue. Claude 3 was best at this, including dead and low-resource languages. Neither modern Claude nor Gemini are remotely close to what Claude 3 was capable of (e.g. zero-shot writing styles). Anthropic basically reversed their "character training" policy and started optimizing their models for code generation at the cost of everything else, starting with Sonnet 3.5. Claude 4 took a huge hit in multilingual ability GPT, on the other hand, was always terrible at languages, except for the short-lived gpt-4.5-preview. All modern models including Gemini have bugs in basic language coherency - random language switching, self-correction attempts resulting in hallucinations etc. I speculate it's a problem with heavy RL with rewards and policies not optimized for creative writing.
- awongh 6mo ago

robocat 6mo ago

> a model that will be an entire generation beyond SOTA That model would then be SOTA. Tautologically you can't be better than SOTA

himata4113 6mo ago
SOTA at that time*

solarkraft 6mo ago

I really wonder what I’m missing with Gemini. It’s a second rate model for me at best. I find it okay (not great) at collecting information and completely useless at agentic tasks. It’s like it’s always drunk. When the Claude credits expire in Antigravity, I’m done for the day. > They produce drastically lower amount of tokens to solve a problem I LOLed at this because I of the constant death loops that don’t even solve the problem at all.

8note 6mo ago
i get much better results with it using a different toolset. give it serena and it mostly works, and is less likely to hit a death loop. i feel like the geminicli app is missing some tools for making sure the session history is actually valid
himata4113 6mo ago
Yah it doesn't even make sense how they got through their benchmarks without death loops. Gemini-cli even has a hotfix to break the model of such death-loops. But if you were to ignore this bug/quirk that will be fixed in the next patch release my point still stands.

nl 6mo ago

> the pro and flash variants are 5x to 10x smaller than opus and gpt-5 class models. The rumor is that Gemini Pro is the largest model being served today (or at least was prior to Mythos) Source: some podcast where they were discussing TPU vs Nvidia cluster topologies, and how Google is exploiting their topology to allow this. But I can't remember exactly which podcast, so hopefully someone else will know.