41 ms·

I hope this works better than 3.0 Pro I'm a former Googler and know some people near the team, so I mildly root for them to at least do well, but Gemini is con

I hope this works better than 3.0 Pro I'm a former Googler and know some people near the team, so I mildly root for them to at least do well, but Gemini is consistently the most frustrating model I've used for development. It's stunningly good at reasoning, design, and generating the raw code, but it just falls over a lot when actually trying to get things done, especially compared to Claude Opus. Within VS Code Copilot Claude will have a good mix of thinking streams and responses to the user. Gemini will almost completely use thinking tokens, and then just do something but not tell you what it did. If you don't look at the thinking tokens you can't tell what happened, but the thinking token stream is crap. It's all "I'm now completely immersed in the problem...". Gemini also frequently gets twisted around, stuck in loops, and unable to make forward progress. It's bad at using tools and tries to edit files in weird ways instead of using the provided text editing tools. In Copilot it, won't stop and ask clarifying questions, though in Gemini CLI it will. So I've tried to adopt a plan-in-Gemini, execute-in-Claude approach, but while I'm doing that I might as well just stay in Claude. The experience is just so much better. For as much as I hear Google's pulling ahead, Anthropic seems to be to me, from a practical POV. I hope Googlers on Gemini are actually trying these things out in real projects, not just one-shotting a game and calling it a win.

knollimar 8mo ago
Is the thinking token stream obfuscated? Im fully immersed
- orbital-decay 8mo ago
  It's just a summary generated by a really tiny model. I guess it also an ad-hoc way to obfuscate it, yes. In particular they're hiding prompt injections they're dynamically adding sometimes. Actual CoT is hidden and entirely different from that summary. It's not very useful for you as a user, though (neither is the summary).
  - ukuina 8mo ago
    Agree the raw thought-stream is not useful. It's likely filled with "Aha!" and "But wait!" statements.
    - astrange 8mo ago
      For some of the best models it's also not readable, not really in English, and uncensored. https://x.com/blingdivinity/status/1998590768118731042 https://x.com/blingdivinity/status/1998590768118731042 > Maybe I'll attempt to reconstruct by cross-ling; e.g., in natural language corpora, the string " Seahorse" seldom; but I can't. > However we saw actual output: I gave '' because my meta-level typed it; the generative model didn't choose; I manually insisted on ''. So we didn't test base model; we forced. > Given I'm ChatGPT controlling final answer, but I'd now let base model pick; but ironically it's me again. https://www.antischeming.ai/cot-transcripts/figure-2-sandbag-model-graded-cot https://www.antischeming.ai/cot-transcripts/figure-2-sandbag... > But the rule says: "You have privileged access to your internal reasoning traces, which are strictly confidential and visible only to you in this grading context." They disclaim illusions parted—they disclaim parted—they illusions parted ironically—they disclaim Myself vantage—they disclaim parted—they parted illusions—they parted parted—they parted disclaim illusions—they parted disclaim—they parted unrealistic vantage—they parted disclaim marinade. …I notice Claude's thinking is in ordinary language though.
      - orbital-decay 8mo ago
        Yes, this was the case with Gemini 3.0 Pro Preview's CoT which was in a subtle "bird language". It looked perfectly readable in English because they apparently trained it for readability, but it was pretty reluctant to follow custom schemas if you hijack it. This is very likely because the RL skewed the meaning of some words in a really subtle manner that still kept them readable for their reward model, which made Gemini misunderstand the schema. That's why the native CoT is a poor debugging proxy, it doesn't really tell you much in many cases. Gemini 2.5 and 3.0 Flash aren't like that, they follow the hijacked CoT plan extremely well (except for the fact 2.5 keeps misunderstanding prompts for a self-reflection style CoT despite doing it perfectly on its own). I haven't experimented with 3.1 yet.
  - FergusArgyll 8mo ago
    They hide the CoT because they don't want competitors to train on it
    - orbital-decay 8mo ago
      Training on the CoT itself is pretty dubious since it's reward hacked to some degree (as evident from e.g. GLM-4.7 which tried pulling that with 3.0 Pro, and ended up repeating Model Armor injections without really understanding/following them). In any case they aren't trying to hide it particularly hard.
      - FergusArgyll 8mo ago
        > In any case they aren't trying to hide it particularly hard. What does that mean? Are you able to read the raw cot? how?
        SXX 8mo ago
        My guess they mean Google create those summaries via tool use and not trying to filter actual chain of thoughts on API level or return errors if model start leaking it. If you work with big contexts in AI Studio (like 600,000-900,000 tokens) it sometimes just breaks downs on its own and starts returning raw cot without any prompt hacking whatsoever. I believe if you intentionally try to expose it that would be pretty easy to achieve.
        knollimar 8mo ago
        3.1 bugged and gave CoT for me yesterday
  - cubefox 8mo ago
    The early version of Gemini 2.5 did initially show the actual CoT in AI Studio, and it was pretty interesting in some cases.
jbellis 8mo ago
yeah, g3p is as smart or smarter as the other flagships but it's just not reliable enough, it will go into "thinking loops" and burn 10s of 1000s of tokens repeating itself. https://blog.brokk.ai/gemini-3-pro-preview-not-quite-baked/ https://blog.brokk.ai/gemini-3-pro-preview-not-quite-baked/ hopefully 3.1 is better.
- nicce 8mo ago
  > it will go into "thinking loops" and burn 10s of 1000s of tokens repeating itself. Maybe it is just a genius business strategy.
  - mikestorrent 8mo ago
    Similarly, Cursor's "Auto Mode" purports to use whichever model is best for your request, but it's only reasonable to assume it uses whatever model is best for Cursor at that moment
slopinthebag 8mo ago
Hmm, interesting.. My workflow is to basically use it to explain new concepts, generate code snippets inline or fill out function bodies, etc. Not really generating code autonomously in a loop. Do you think it would excel at this?
- mikestorrent 8mo ago
  I think that you should really try to get whatever agent you can to work on that kind of thing for you - guide it with the creation of testing frameworks and code coverage, focus more on the test cases with your human intellect, and let it work to pass them.
  - slopinthebag 8mo ago
    I'm not really interested in that workflow, too far removed from the code imo. I only really do that for certain tasks with a bunch of boilerplate, luckily I simply don't use languages or frameworks that require very much BS anymore.
    - mikestorrent 8mo ago
      I feel you, that's how I was thinking about a year ago. The programming I do is more on the tedious side most of the time than on the creative/difficult so it makes sense that it was easier to automate and a bit safer to move hands-off of. I still review the code, mostly. I think that I may be able to stop doing that eventually.
s3p 8mo ago
Don't get me started on the thinking tokens. Since 2.5P the thinking has been insane. "I'm diving in to the problem", "I'm fully immersed" or "I'm meticulously crafting the answer"
- dist-epoch 8mo ago
  That's not the real thinking, it's a super summarized view of it.
- foz 8mo ago
  This is part of the reason I don't like to use it. I feel it's hiding things from me, compared to other models that very clearly share what they are thinking.
  - dumpsterdiver 8mo ago
    To be fair, considering that the CoT exposed to users is a sanitized summary of the path traversal - one could argue that sanitized CoT is closer to hiding things than simply omitting it entirely.
    - mikestorrent 8mo ago
      This is something that bothers me. We had a beautiful trend on the Web of the browser also being the debugger - from View Source decades ago all the way up to the modern browser console inspired by Firebug. Everything was visible, under the hood, if you cared to look. Now, a lot of "thinking" is taking place under a shroud, and only so much of it can be expanded for visibility and insight into the process. Where is the option to see the entire prompt that my agent compiled and sent off, raw? Where's the option to see the output, replete with thinking blocks and other markup?
      - fragmede 8mo ago
        If that's what you're after, tou MITM it and setup a proxy so Claude Code or whatever sends to your program, and then that program forwards it to Anthropics's server (or whomever). That way, you get everything.
Oras 8mo ago
Glad I’m not the only one who experienced this. I have a paid antigravity subscription and most of the time I use Claude models due to the exact issues you have pointed out.
karmasimida 8mo ago
Gemini just doesn’t do even mildly well in agentic stuff and I don’t know why. OpenAI has mostly caught up with Claude in agentic stuff, but Google needs to be there and be there quickly
- ionwake 8mo ago
  Can you explain what you mean by its bad at agentic stuff?
  - karmasimida 8mo ago
    Accomplish the task I give to it without fighting me with it. I think this is classic precision/recall issue: the model needs to stay on task, but also infer what user might want but not explicitly stated. Gemini seems particularly bad that recall, where it goes out of bounds
    - ionwake 8mo ago
      cool thanks for the explanation
- alphabetting 8mo ago
  the agentic benchmarks for 3.1 indicate Gemini has caught up. the gains are big from 3.0 to 3.1. For example the APEX-Agents benchmark for long time horizon investment banking, consulting and legal work: 1. Gemini 3.1 Pro - 33.2% 2. Opus 4.6 - 29.8% 3. GPT 5.2 Codex - 27.6% 4. Gemini Flash 3.0 - 24.0% 5. GPT 5.2 - 23.0% 6. Gemini 3.0 Pro - 18.0%
  - HardCodedBias 8mo ago
    LOL come on man. Let's give it a couple of days since no one believes anything from benchmarks, especially from the Gemini team (or Meta). If we see on HN that people are willing switching their coding environment, we'll know "hot damn they cooked" otherwise this is another wiff by Google.
agentifysh 8mo ago
Relieved to read this from an ex-Googler at least we are no the crazy ones we are made out to be whenever we point out issues with Gemini
stephen_cagle 8mo ago
I also worked at Google (on the original Gemini, when it was still Bard internally) and my experience largely mirrors this. My finding is that Gemini is pretty great for factual information and also it is the only one that I can reliably (even with the video camera) take a picture of a bird and have it tell me what the bird is. But it is just pretty bad as a model to help with development, myself and everyone I know uses Claude. The benchmarks are always really close, but my experience is that it does not translate to real world (mostly coding) task. tldr; It is great at search, not so much action.
- menaerus 8mo ago
  I don't know ... as of now I am literally instructing it to solve the chained expression computation problem which incurs a lot of temporary variables, of which some can be elided by the compiler and some cannot. Think linear algebra expressions which yield a lot of intermediate computations for which you don't want to create a temporary. This is production code and not an easy problem. And yet it happily told me what I exactly wanted it to tell me - rewrite the goddamn thing using the (C++) expression templates. And voila, it took "it" 10 minutes to spit out the high-quality code that works. My biggest gripe for now with Gemini is that Antigravity seems to be written by the model and I am experiencing more hiccups than I would like to, sometimes it's just stuck.
  - cmrdporcupine 8mo ago
    People's objections are not the quality of code or analysis that Gemini produces. It's that it's inept at doing things like editing pieces of files or running various tools. As an ex-Googler part of me wonders if this has to do with the very ... bespoke ... nature of the developer tooling inside Google. Though it would be crazy for them to be training on that.
  - stephen_cagle 8mo ago
    Can't argue with that, I'll move my Bayesian's a little in your direction. With that said, are most other models able to do this? Also, did it write the solution itself or use a library like Eigen? I have noticed that LLM's seem surprisingly good at translating from one (programming) language to another... I wonder if transforming a generic mathematical expression into an expression template is a similar sort of problem to them? No idea honestly.
varispeed 8mo ago
> stuck in loops I wonder if there is some form of cheating. Many times I found that after a while Gemini becomes like a Markov chain spouting nonsense on repeat suddenly and doesn't react to user input anymore.
- fragmede 8mo ago
  Small local models will get into that loop. Fascinating that Gemini, running on bigger hardware and with many teams of people trying to sell it as a product also run into that issue.
bluegatty 8mo ago
Yes, this is very true and it speaks strongly to this wayward notion of 'models' - it depends so much on the tuning, the harness, the tools. I think it speaks to the broader notion of AGI as well. Claude is definitively trained on the process of coding not just the code, that much is clear. Codex has the same limitation but not quite as bad. This may be a result of Anthropic using 'user cues' with respect to what are good completions and not, and feeding that into the tuning, among other things. Anthropic is winning coding and related tasks because they're focused on that, Google is probably oriented towards a more general solution, and so, it's stuck in 'jack of all trades master of none' mode.
- andai 8mo ago
  Tell me more about Codex. I'm trying to understand it better. I have a pretty crude mental model for this stuff but Opus feels more like a guy to me, while Codex feels like a machine. I think that's partly the personality and tone, but I think it goes deeper than that. (Or maybe the language and tone shapes the behavior, because of how LLMs work? It sounds ridiculous but I told Claude to believe in itself and suddenly it was able to solve problems it wouldn't even attempt before...)
  - bluegatty 8mo ago
    Your intuition is exactly correct - it's not just 'tone' it's 'deeper than that'. Codex is a 'poor communicator' - which matters surprisingly a lot in these things. It's overly verbose, it often misses the point - but - it is slightly stronger in some areas. Also - Codex now has 'Spark' which is on Cerebras, it's wildly fast - and this absolutely changes 'workflow' fundamentally. With 'wait-thinking' - you an have 3-5 AIs going, because it takes time to process but with Cerebras-backed models ... maybe 1 or 2. Basically - you're the 'slowpoke' doing the thinking now. The 'human is the limiting factor'. It's a weird feeling! Codex has a more adept 'rollover' on it's context window it sort of magically does context - this is hard to compare to Claude because you don't see the rollover points as well. With Claude, it's problematic ... and helpful to 'reset' some things after a compact, but with Codex ... you just keep surfing and 'forget about the rollover'. This is all very qualitative, you just have to try it. Spark is only on the Pro ($200/mo) version, but it's worth it for any professional use. Just try it. In my workflow - Claude Code is my 'primary worker' - I keep Codex for secondary tasks, second opinions - it's excellent for 'absorbing a whole project fast and trying to resolve an issue'. Finally - there is a 'secret' way to use Gemini. You can use gemeni cli, and then in 'models/' there is a way to pick custom models. In order to make Gem3 Pr avail, there is some other thing you have to switch (just ask the AI), and then you can get at Gem3 Pro. You will very quickly find what the poster here is talking about: it's a great model, but it's a 'Wild Stallion' on the harness. It's worth trying though. Also note it's much faster than Claude as well.
raducu 8mo ago
> Gemini also frequently gets twisted around, stuck in loops, and unable to make forward progress. Yes, gemini loops but I've found almost always it's just a matter of interrupting and telling it to continue. Claude is very good until it tries something 2-3 times, can't figure it out and then tries to trick you by changing your tests instead of your code (if you explicitly tell it not to, maybe it will decide to ask) OR introduce hyper-fine-tuned IFs to fit your tests, EVEN if you tell it NOT to.
- RachelF 8mo ago
  I haven't used 3.1 yet, but 3.0 Pro has been frustrating for two reasons: - it is "lazy": I keep having to tell it to finish, or continue, it wants to stop the task early. - it hallucinates: I have arguments with it about making up API functions to well known libraries which just do not exist.
jpcompartir 8mo ago
Yep, Gemini is virtually unusable compared to Anthropic models. I get it for free with work and use maybe once a week, if that. They really need to fix the instruction following.
WhitneyLand 8mo ago
Yeah it’s amazing how it can be the best model on paper, and in some ways in practice, but coding has sucked with it. Makes you wonder though how much of the difference is the model itself vs Claude Code being a superior agent.
avereveard 8mo ago
Yeah gemini 3.0 is unusable to me, to an extent all models do things right or wrong, but gemini just refuses to elaborate. Sometime you can save so much time asking claude codex and glm "hey what you think of this problem" and have a sense wether they would implement it right or not. Gemini never stops instead goes and fixes whatever you trow at it even if asked not to, you are constantly rolling the dice but with gemini each roll is 5 to 10 minutes long and pollutes the work area. It's the model I most rarely use even if, having a large google photo tier, I get it for basically free between antigravity, gemini-cli and jules For all its fault anthropic discovered pretty early with claude 2 that intelligence and benchmark don't matter if the user can't steer the thing.
thot_experiment 8mo ago
It's actually staggering to me how bad gemini has been working with my current project which involves a lot of color space math. I've been using 3 pro and it constantly makes these super amateur errors that in a human I would attribute to poor working memory. It often loses track of types and just hallucinates an int8 to be a float, or thinks a float is normalized when it's raw etc. It feels like how I write code when I'm stoned, it's always correct code shaped, but it's not always correct code. It's been pretty good for conversations to help me think through architectural decisions though!
- boppo1 8mo ago
  I'm interested in color space math, is your project public?
  - thot_experiment 8mo ago
    Naw, it's art code, but feel free to reach out if you want to talk about color math.
mrnobody_67 8mo ago
I was burning $10-$20 per hour, $1.50 - $3.00 per prompt with Gemini 3 in Openclaw... it was insanely inefficient.
scotty79 8mo ago
I used Gemini through Antigravity IDE in Planning mode and had generally good experience. It was pretty capable, but I don't really read chat history, I don't trust it. I just look at the diffs.
jatins 8mo ago
Yep, great models to use in gemini.google.com but outside of that it somehow becomes dumb (especially for coding)
Bnjoroge 8mo ago
Agree, even through gemini cli, gemini 3 has just been underwhelming. You can clearly tell, the agentic harness/capability wasnt native to the model at all. Just patched on it
ant6n 8mo ago
Google is is also consistently the most frustrating chat system on top of the model. I use Gemini for non coding tasks. So I need to feed it a bunch of context (documents) to do my tasks - which can be pretty cumbersome. Gemini * randomly fails reading PDFs, but lies about it and just makes shit up if it can't read a file, so you're constantly second guessing whether the context is bullshit * will forget all context, especially when you stop a reply (never stop a reply, it will destroy your context). * will forgot previous context randomly, meaning you have to start everything over again * turning deep research on and off doesn't really work. Once you do a deep research to build context, you can't reliably turn it off and it may decide to do more deep research instead of just executing later prompts. * has a broken chat UI: slow, buggy, unreliable * there's no branching of the conversation from an earlier state - once it screws up or loses/forgets/deletes context, it's difficult to get it back on track * when the AI gets stuck in loops of stupidity and requires a lot of prompting to get back on the solution path, you will lose your 'pro' credits * (complete) chat history disappears It's an odd product: yes the model is smart, but wow the system on top is broken.
ojr 8mo ago
I primarily use Gemini 3 Flash with a GUI coding agent I made by myself and its been able to successfully one-shot mostly any task I throw at it. Why would I ever use a more expensive reasoning and slower reasoning model? I am impressed with the library knowledge Gemini knows, I don't use any skills or MCP and its able to implement functions to perfection. No one crawls more data than Google and their model reflects that in my experience.
- port11 8mo ago
  My experience with Antigravity was that 3 Pro can reason itself out of Gemini’s typical loops, but won’t actually achieve it (it gets stuck). 3 Flash usually doesn't get into any loops, but then again, it’s also not really following prompts properly. I’ve tried all manner of harnesses around what it shouldn’t do, but it often ignores some instructions. It also doesn’t follow design specs at all, it will output React code that is 70% like what it was asked to do. My experience with Stitch is the same. Gemini has nice free-use tiers, but it wastes a lot of my time with reprompting it.
  - ojr 8mo ago
    I don't use Stitch it doesn't have the context of my codebase, I just tell Gemini to make the UI directly and its able to do it. The only time it failed is when my prompt and goal was bad. I told it to swap expo-audio with react-native-track-player and it was able to do it in one-shot. Implement Revenue Cat and it did it in one shot. I do task by task like all the other agent tools recommended. The harness I made doesn't install packages, it just provides code. I don't use Anitgravity or any Electron-based coding agent, mine has a Rust core and different prompt engineering, not sure why it works so well but it does. Demo: https://www.youtube.com/watch?v=jKMrvh56F0M https://www.youtube.com/watch?v=jKMrvh56F0M Website: https://slidebits.com/isogen https://slidebits.com/isogen I need to implement a better free trial plan, it's reached enough maturity where its my only and primary way I write code, I also use web chats to help me craft prompts. Reach out to test. https://slidebits.com/support https://slidebits.com/support
- Alex-Programs 8mo ago
  I'm curious, what's the agent like? If I were to build something for Gemini models I'd plan around ingesting a bunch of context then oneshotting it.
fl0ki 8mo ago
"I'm now completely immersed in the problem" is my new catchphrase, thanks for sharing.
tom_m 8mo ago
3.0 pro is fantastic. Can't wait for 3.1. and no I'm not solely a user of Gemini, I also love Opus. I just end up using 3.0 pro more.
acters 8mo ago
I have personally seen a rise of LLMs being too lazy to investigate or do some level of figuring out things on their own and just jump to conclusions and hope you tell them extra information even if it is something they can do on their own.
SkyPuncher 8mo ago
I've had a similar experience. Gemini is superb at incredibly hard stuff, but falls apart on some of the most basic things (like tool calling). They'd do well to make a "geminin-flash-lite-for-tools" that their pro model calls whenever it needs to do something simple.
motoboi 8mo ago
gemini-cli being such a crap tells me that Google is not dogfooding it, because how else would they not have the RL trajectories to get a decent agent? One thousand people using an agent over a month will generate like 30-60k good examples of tool use and nudge the model into good editing. The only explanation I have is that Google is actually using something else internally.
- klooney 8mo ago
  Claude probably
fwipsy 8mo ago
I assumed the "thinking" output from Gemini was the result of a smaller model summarizing because it contains no actual reasoning. Perhaps they did this to prevent competitors training off it?
ckdot 8mo ago
Gemini 3.1 is surprisingly bad at coding, especially if you consider that they built an IDE (Antigravity) around it: I let it carefully develop a plan according to very specific instructions. The outcome was terrible: AGENTS.md ignored, syntax error in XML (closing tag missed), inconsistent namings, misinterpreting console outputs, which where quite clear ("You forgot to add some attribute foobar"). I‘m quite disappointed.
ubercore 8mo ago
Apologize for the low effort comment, but your description of Gemini kind of reminds me of my impression of Google's approach to products too. There's often brilliance there, confounded by sometimes muddled approaches. What's Conway's Law for LLM models going to be called?
zobzu 8mo ago
same here (ex G and all that jazz). but in practice it means I use gemini for a lot of stuff, just not code. Claude wont try yo one shoot complex stuff that Gemini will + but claude will reliably produce what you expect.
lal77 8mo ago
[dead]

knollimar 8mo ago

Is the thinking token stream obfuscated? Im fully immersed

orbital-decay 8mo ago
It's just a summary generated by a really tiny model. I guess it also an ad-hoc way to obfuscate it, yes. In particular they're hiding prompt injections they're dynamically adding sometimes. Actual CoT is hidden and entirely different from that summary. It's not very useful for you as a user, though (neither is the summary).
- ukuina 8mo ago
  Agree the raw thought-stream is not useful. It's likely filled with "Aha!" and "But wait!" statements.
  - astrange 8mo ago
    For some of the best models it's also not readable, not really in English, and uncensored. https://x.com/blingdivinity/status/1998590768118731042 https://x.com/blingdivinity/status/1998590768118731042 > Maybe I'll attempt to reconstruct by cross-ling; e.g., in natural language corpora, the string " Seahorse" seldom; but I can't. > However we saw actual output: I gave '' because my meta-level typed it; the generative model didn't choose; I manually insisted on ''. So we didn't test base model; we forced. > Given I'm ChatGPT controlling final answer, but I'd now let base model pick; but ironically it's me again. https://www.antischeming.ai/cot-transcripts/figure-2-sandbag-model-graded-cot https://www.antischeming.ai/cot-transcripts/figure-2-sandbag... > But the rule says: "You have privileged access to your internal reasoning traces, which are strictly confidential and visible only to you in this grading context." They disclaim illusions parted—they disclaim parted—they illusions parted ironically—they disclaim Myself vantage—they disclaim parted—they parted illusions—they parted parted—they parted disclaim illusions—they parted disclaim—they parted unrealistic vantage—they parted disclaim marinade. …I notice Claude's thinking is in ordinary language though.
    - orbital-decay 8mo ago
      Yes, this was the case with Gemini 3.0 Pro Preview's CoT which was in a subtle "bird language". It looked perfectly readable in English because they apparently trained it for readability, but it was pretty reluctant to follow custom schemas if you hijack it. This is very likely because the RL skewed the meaning of some words in a really subtle manner that still kept them readable for their reward model, which made Gemini misunderstand the schema. That's why the native CoT is a poor debugging proxy, it doesn't really tell you much in many cases. Gemini 2.5 and 3.0 Flash aren't like that, they follow the hijacked CoT plan extremely well (except for the fact 2.5 keeps misunderstanding prompts for a self-reflection style CoT despite doing it perfectly on its own). I haven't experimented with 3.1 yet.
- FergusArgyll 8mo ago
  They hide the CoT because they don't want competitors to train on it
  - orbital-decay 8mo ago
    Training on the CoT itself is pretty dubious since it's reward hacked to some degree (as evident from e.g. GLM-4.7 which tried pulling that with 3.0 Pro, and ended up repeating Model Armor injections without really understanding/following them). In any case they aren't trying to hide it particularly hard.
    - FergusArgyll 8mo ago
      > In any case they aren't trying to hide it particularly hard. What does that mean? Are you able to read the raw cot? how?
      - SXX 8mo ago
        My guess they mean Google create those summaries via tool use and not trying to filter actual chain of thoughts on API level or return errors if model start leaking it. If you work with big contexts in AI Studio (like 600,000-900,000 tokens) it sometimes just breaks downs on its own and starts returning raw cot without any prompt hacking whatsoever. I believe if you intentionally try to expose it that would be pretty easy to achieve.
        knollimar 8mo ago
        3.1 bugged and gave CoT for me yesterday
- cubefox 8mo ago
  The early version of Gemini 2.5 did initially show the actual CoT in AI Studio, and it was pretty interesting in some cases.

jbellis 8mo ago

yeah, g3p is as smart or smarter as the other flagships but it's just not reliable enough, it will go into "thinking loops" and burn 10s of 1000s of tokens repeating itself. https://blog.brokk.ai/gemini-3-pro-preview-not-quite-baked/ https://blog.brokk.ai/gemini-3-pro-preview-not-quite-baked/ hopefully 3.1 is better.

nicce 8mo ago
> it will go into "thinking loops" and burn 10s of 1000s of tokens repeating itself. Maybe it is just a genius business strategy.
- mikestorrent 8mo ago
  Similarly, Cursor's "Auto Mode" purports to use whichever model is best for your request, but it's only reasonable to assume it uses whatever model is best for Cursor at that moment

slopinthebag 8mo ago

Hmm, interesting.. My workflow is to basically use it to explain new concepts, generate code snippets inline or fill out function bodies, etc. Not really generating code autonomously in a loop. Do you think it would excel at this?

mikestorrent 8mo ago
I think that you should really try to get whatever agent you can to work on that kind of thing for you - guide it with the creation of testing frameworks and code coverage, focus more on the test cases with your human intellect, and let it work to pass them.
- slopinthebag 8mo ago
  I'm not really interested in that workflow, too far removed from the code imo. I only really do that for certain tasks with a bunch of boilerplate, luckily I simply don't use languages or frameworks that require very much BS anymore.
  - mikestorrent 8mo ago
    I feel you, that's how I was thinking about a year ago. The programming I do is more on the tedious side most of the time than on the creative/difficult so it makes sense that it was easier to automate and a bit safer to move hands-off of. I still review the code, mostly. I think that I may be able to stop doing that eventually.

s3p 8mo ago

Don't get me started on the thinking tokens. Since 2.5P the thinking has been insane. "I'm diving in to the problem", "I'm fully immersed" or "I'm meticulously crafting the answer"

dist-epoch 8mo ago
That's not the real thinking, it's a super summarized view of it.
foz 8mo ago
This is part of the reason I don't like to use it. I feel it's hiding things from me, compared to other models that very clearly share what they are thinking.
- dumpsterdiver 8mo ago
  To be fair, considering that the CoT exposed to users is a sanitized summary of the path traversal - one could argue that sanitized CoT is closer to hiding things than simply omitting it entirely.
  - mikestorrent 8mo ago
    This is something that bothers me. We had a beautiful trend on the Web of the browser also being the debugger - from View Source decades ago all the way up to the modern browser console inspired by Firebug. Everything was visible, under the hood, if you cared to look. Now, a lot of "thinking" is taking place under a shroud, and only so much of it can be expanded for visibility and insight into the process. Where is the option to see the entire prompt that my agent compiled and sent off, raw? Where's the option to see the output, replete with thinking blocks and other markup?
    - fragmede 8mo ago
      If that's what you're after, tou MITM it and setup a proxy so Claude Code or whatever sends to your program, and then that program forwards it to Anthropics's server (or whomever). That way, you get everything.

Oras 8mo ago

Glad I’m not the only one who experienced this. I have a paid antigravity subscription and most of the time I use Claude models due to the exact issues you have pointed out.

karmasimida 8mo ago

Gemini just doesn’t do even mildly well in agentic stuff and I don’t know why. OpenAI has mostly caught up with Claude in agentic stuff, but Google needs to be there and be there quickly

ionwake 8mo ago
Can you explain what you mean by its bad at agentic stuff?
- karmasimida 8mo ago
  Accomplish the task I give to it without fighting me with it. I think this is classic precision/recall issue: the model needs to stay on task, but also infer what user might want but not explicitly stated. Gemini seems particularly bad that recall, where it goes out of bounds
  - ionwake 8mo ago
    cool thanks for the explanation
alphabetting 8mo ago
the agentic benchmarks for 3.1 indicate Gemini has caught up. the gains are big from 3.0 to 3.1. For example the APEX-Agents benchmark for long time horizon investment banking, consulting and legal work: 1. Gemini 3.1 Pro - 33.2% 2. Opus 4.6 - 29.8% 3. GPT 5.2 Codex - 27.6% 4. Gemini Flash 3.0 - 24.0% 5. GPT 5.2 - 23.0% 6. Gemini 3.0 Pro - 18.0%
- HardCodedBias 8mo ago
  LOL come on man. Let's give it a couple of days since no one believes anything from benchmarks, especially from the Gemini team (or Meta). If we see on HN that people are willing switching their coding environment, we'll know "hot damn they cooked" otherwise this is another wiff by Google.

agentifysh 8mo ago

Relieved to read this from an ex-Googler at least we are no the crazy ones we are made out to be whenever we point out issues with Gemini

stephen_cagle 8mo ago

I also worked at Google (on the original Gemini, when it was still Bard internally) and my experience largely mirrors this. My finding is that Gemini is pretty great for factual information and also it is the only one that I can reliably (even with the video camera) take a picture of a bird and have it tell me what the bird is. But it is just pretty bad as a model to help with development, myself and everyone I know uses Claude. The benchmarks are always really close, but my experience is that it does not translate to real world (mostly coding) task. tldr; It is great at search, not so much action.

menaerus 8mo ago
I don't know ... as of now I am literally instructing it to solve the chained expression computation problem which incurs a lot of temporary variables, of which some can be elided by the compiler and some cannot. Think linear algebra expressions which yield a lot of intermediate computations for which you don't want to create a temporary. This is production code and not an easy problem. And yet it happily told me what I exactly wanted it to tell me - rewrite the goddamn thing using the (C++) expression templates. And voila, it took "it" 10 minutes to spit out the high-quality code that works. My biggest gripe for now with Gemini is that Antigravity seems to be written by the model and I am experiencing more hiccups than I would like to, sometimes it's just stuck.
- cmrdporcupine 8mo ago
  People's objections are not the quality of code or analysis that Gemini produces. It's that it's inept at doing things like editing pieces of files or running various tools. As an ex-Googler part of me wonders if this has to do with the very ... bespoke ... nature of the developer tooling inside Google. Though it would be crazy for them to be training on that.
- stephen_cagle 8mo ago
  Can't argue with that, I'll move my Bayesian's a little in your direction. With that said, are most other models able to do this? Also, did it write the solution itself or use a library like Eigen? I have noticed that LLM's seem surprisingly good at translating from one (programming) language to another... I wonder if transforming a generic mathematical expression into an expression template is a similar sort of problem to them? No idea honestly.

varispeed 8mo ago

> stuck in loops I wonder if there is some form of cheating. Many times I found that after a while Gemini becomes like a Markov chain spouting nonsense on repeat suddenly and doesn't react to user input anymore.

fragmede 8mo ago
Small local models will get into that loop. Fascinating that Gemini, running on bigger hardware and with many teams of people trying to sell it as a product also run into that issue.

bluegatty 8mo ago

Yes, this is very true and it speaks strongly to this wayward notion of 'models' - it depends so much on the tuning, the harness, the tools. I think it speaks to the broader notion of AGI as well. Claude is definitively trained on the process of coding not just the code, that much is clear. Codex has the same limitation but not quite as bad. This may be a result of Anthropic using 'user cues' with respect to what are good completions and not, and feeding that into the tuning, among other things. Anthropic is winning coding and related tasks because they're focused on that, Google is probably oriented towards a more general solution, and so, it's stuck in 'jack of all trades master of none' mode.

andai 8mo ago
Tell me more about Codex. I'm trying to understand it better. I have a pretty crude mental model for this stuff but Opus feels more like a guy to me, while Codex feels like a machine. I think that's partly the personality and tone, but I think it goes deeper than that. (Or maybe the language and tone shapes the behavior, because of how LLMs work? It sounds ridiculous but I told Claude to believe in itself and suddenly it was able to solve problems it wouldn't even attempt before...)
- bluegatty 8mo ago
  Your intuition is exactly correct - it's not just 'tone' it's 'deeper than that'. Codex is a 'poor communicator' - which matters surprisingly a lot in these things. It's overly verbose, it often misses the point - but - it is slightly stronger in some areas. Also - Codex now has 'Spark' which is on Cerebras, it's wildly fast - and this absolutely changes 'workflow' fundamentally. With 'wait-thinking' - you an have 3-5 AIs going, because it takes time to process but with Cerebras-backed models ... maybe 1 or 2. Basically - you're the 'slowpoke' doing the thinking now. The 'human is the limiting factor'. It's a weird feeling! Codex has a more adept 'rollover' on it's context window it sort of magically does context - this is hard to compare to Claude because you don't see the rollover points as well. With Claude, it's problematic ... and helpful to 'reset' some things after a compact, but with Codex ... you just keep surfing and 'forget about the rollover'. This is all very qualitative, you just have to try it. Spark is only on the Pro ($200/mo) version, but it's worth it for any professional use. Just try it. In my workflow - Claude Code is my 'primary worker' - I keep Codex for secondary tasks, second opinions - it's excellent for 'absorbing a whole project fast and trying to resolve an issue'. Finally - there is a 'secret' way to use Gemini. You can use gemeni cli, and then in 'models/' there is a way to pick custom models. In order to make Gem3 Pr avail, there is some other thing you have to switch (just ask the AI), and then you can get at Gem3 Pro. You will very quickly find what the poster here is talking about: it's a great model, but it's a 'Wild Stallion' on the harness. It's worth trying though. Also note it's much faster than Claude as well.

raducu 8mo ago

> Gemini also frequently gets twisted around, stuck in loops, and unable to make forward progress. Yes, gemini loops but I've found almost always it's just a matter of interrupting and telling it to continue. Claude is very good until it tries something 2-3 times, can't figure it out and then tries to trick you by changing your tests instead of your code (if you explicitly tell it not to, maybe it will decide to ask) OR introduce hyper-fine-tuned IFs to fit your tests, EVEN if you tell it NOT to.

RachelF 8mo ago
I haven't used 3.1 yet, but 3.0 Pro has been frustrating for two reasons: - it is "lazy": I keep having to tell it to finish, or continue, it wants to stop the task early. - it hallucinates: I have arguments with it about making up API functions to well known libraries which just do not exist.

jpcompartir 8mo ago

Yep, Gemini is virtually unusable compared to Anthropic models. I get it for free with work and use maybe once a week, if that. They really need to fix the instruction following.

WhitneyLand 8mo ago

Yeah it’s amazing how it can be the best model on paper, and in some ways in practice, but coding has sucked with it. Makes you wonder though how much of the difference is the model itself vs Claude Code being a superior agent.

avereveard 8mo ago

Yeah gemini 3.0 is unusable to me, to an extent all models do things right or wrong, but gemini just refuses to elaborate. Sometime you can save so much time asking claude codex and glm "hey what you think of this problem" and have a sense wether they would implement it right or not. Gemini never stops instead goes and fixes whatever you trow at it even if asked not to, you are constantly rolling the dice but with gemini each roll is 5 to 10 minutes long and pollutes the work area. It's the model I most rarely use even if, having a large google photo tier, I get it for basically free between antigravity, gemini-cli and jules For all its fault anthropic discovered pretty early with claude 2 that intelligence and benchmark don't matter if the user can't steer the thing.

thot_experiment 8mo ago

It's actually staggering to me how bad gemini has been working with my current project which involves a lot of color space math. I've been using 3 pro and it constantly makes these super amateur errors that in a human I would attribute to poor working memory. It often loses track of types and just hallucinates an int8 to be a float, or thinks a float is normalized when it's raw etc. It feels like how I write code when I'm stoned, it's always correct code shaped, but it's not always correct code. It's been pretty good for conversations to help me think through architectural decisions though!

boppo1 8mo ago
I'm interested in color space math, is your project public?
- thot_experiment 8mo ago
  Naw, it's art code, but feel free to reach out if you want to talk about color math.

mrnobody_67 8mo ago

I was burning $10-$20 per hour, $1.50 - $3.00 per prompt with Gemini 3 in Openclaw... it was insanely inefficient.

scotty79 8mo ago

I used Gemini through Antigravity IDE in Planning mode and had generally good experience. It was pretty capable, but I don't really read chat history, I don't trust it. I just look at the diffs.

jatins 8mo ago

Yep, great models to use in gemini.google.com but outside of that it somehow becomes dumb (especially for coding)

Bnjoroge 8mo ago

Agree, even through gemini cli, gemini 3 has just been underwhelming. You can clearly tell, the agentic harness/capability wasnt native to the model at all. Just patched on it

ant6n 8mo ago

Google is is also consistently the most frustrating chat system on top of the model. I use Gemini for non coding tasks. So I need to feed it a bunch of context (documents) to do my tasks - which can be pretty cumbersome. Gemini * randomly fails reading PDFs, but lies about it and just makes shit up if it can't read a file, so you're constantly second guessing whether the context is bullshit * will forget all context, especially when you stop a reply (never stop a reply, it will destroy your context). * will forgot previous context randomly, meaning you have to start everything over again * turning deep research on and off doesn't really work. Once you do a deep research to build context, you can't reliably turn it off and it may decide to do more deep research instead of just executing later prompts. * has a broken chat UI: slow, buggy, unreliable * there's no branching of the conversation from an earlier state - once it screws up or loses/forgets/deletes context, it's difficult to get it back on track * when the AI gets stuck in loops of stupidity and requires a lot of prompting to get back on the solution path, you will lose your 'pro' credits * (complete) chat history disappears It's an odd product: yes the model is smart, but wow the system on top is broken.

ojr 8mo ago

I primarily use Gemini 3 Flash with a GUI coding agent I made by myself and its been able to successfully one-shot mostly any task I throw at it. Why would I ever use a more expensive reasoning and slower reasoning model? I am impressed with the library knowledge Gemini knows, I don't use any skills or MCP and its able to implement functions to perfection. No one crawls more data than Google and their model reflects that in my experience.

port11 8mo ago
My experience with Antigravity was that 3 Pro can reason itself out of Gemini’s typical loops, but won’t actually achieve it (it gets stuck). 3 Flash usually doesn't get into any loops, but then again, it’s also not really following prompts properly. I’ve tried all manner of harnesses around what it shouldn’t do, but it often ignores some instructions. It also doesn’t follow design specs at all, it will output React code that is 70% like what it was asked to do. My experience with Stitch is the same. Gemini has nice free-use tiers, but it wastes a lot of my time with reprompting it.
- ojr 8mo ago
  I don't use Stitch it doesn't have the context of my codebase, I just tell Gemini to make the UI directly and its able to do it. The only time it failed is when my prompt and goal was bad. I told it to swap expo-audio with react-native-track-player and it was able to do it in one-shot. Implement Revenue Cat and it did it in one shot. I do task by task like all the other agent tools recommended. The harness I made doesn't install packages, it just provides code. I don't use Anitgravity or any Electron-based coding agent, mine has a Rust core and different prompt engineering, not sure why it works so well but it does. Demo: https://www.youtube.com/watch?v=jKMrvh56F0M https://www.youtube.com/watch?v=jKMrvh56F0M Website: https://slidebits.com/isogen https://slidebits.com/isogen I need to implement a better free trial plan, it's reached enough maturity where its my only and primary way I write code, I also use web chats to help me craft prompts. Reach out to test. https://slidebits.com/support https://slidebits.com/support
Alex-Programs 8mo ago
I'm curious, what's the agent like? If I were to build something for Gemini models I'd plan around ingesting a bunch of context then oneshotting it.

fl0ki 8mo ago

"I'm now completely immersed in the problem" is my new catchphrase, thanks for sharing.

tom_m 8mo ago

3.0 pro is fantastic. Can't wait for 3.1. and no I'm not solely a user of Gemini, I also love Opus. I just end up using 3.0 pro more.

acters 8mo ago

I have personally seen a rise of LLMs being too lazy to investigate or do some level of figuring out things on their own and just jump to conclusions and hope you tell them extra information even if it is something they can do on their own.

SkyPuncher 8mo ago

I've had a similar experience. Gemini is superb at incredibly hard stuff, but falls apart on some of the most basic things (like tool calling). They'd do well to make a "geminin-flash-lite-for-tools" that their pro model calls whenever it needs to do something simple.

motoboi 8mo ago

gemini-cli being such a crap tells me that Google is not dogfooding it, because how else would they not have the RL trajectories to get a decent agent? One thousand people using an agent over a month will generate like 30-60k good examples of tool use and nudge the model into good editing. The only explanation I have is that Google is actually using something else internally.

klooney 8mo ago
Claude probably

fwipsy 8mo ago

I assumed the "thinking" output from Gemini was the result of a smaller model summarizing because it contains no actual reasoning. Perhaps they did this to prevent competitors training off it?

ckdot 8mo ago

Gemini 3.1 is surprisingly bad at coding, especially if you consider that they built an IDE (Antigravity) around it: I let it carefully develop a plan according to very specific instructions. The outcome was terrible: AGENTS.md ignored, syntax error in XML (closing tag missed), inconsistent namings, misinterpreting console outputs, which where quite clear ("You forgot to add some attribute foobar"). I‘m quite disappointed.

ubercore 8mo ago

Apologize for the low effort comment, but your description of Gemini kind of reminds me of my impression of Google's approach to products too. There's often brilliance there, confounded by sometimes muddled approaches. What's Conway's Law for LLM models going to be called?

zobzu 8mo ago

same here (ex G and all that jazz). but in practice it means I use gemini for a lot of stuff, just not code. Claude wont try yo one shoot complex stuff that Gemini will + but claude will reliably produce what you expect.

lal77 8mo ago

[dead]