6 ms·

At one point, when they made Gemini Pro free on AI Studio, Gemini was the model of choice for many people, I believe. Somehow it's gotten worse since then, and

At one point, when they made Gemini Pro free on AI Studio, Gemini was the model of choice for many people, I believe. Somehow it's gotten worse since then, and I'm back to using Claude for serious work. Gemini is like that guy who keeps talking but has no idea what he's actually talking about. I still use Gemini for brainstorming, though I take its suggestions with several grains of salt. It's also useful for generating prompts that I can then refine and use with Claude.

chrismustcode 1y ago
When I ask it do to do something in cursor it goes full sherlock thinking about every possible outcome. Just claude 4 sonnet with thinking just has a bit think then does it
UncleOxidant 1y ago
Used to be able to use Gemini Pro free in cline. Now the API limits are so low that you immediately get messages about needing to top up your wallet and API queries just don't go through. Back to using DeepSeek R1 free in cline (though even that eventually stops after a few hours and you have to wait until the next day for it to work again). Starting to look like I need to setup a local LLM for coding - which means it's time to seriously upgrade my PC (well, it's been about 10 years so it was getting to be time anyway)
- Workaccount2 1y ago
  By the time you breakeven on whatever you spend on a decent LLM capable build, your hardware will be too far behind to run whatever is best locally then. It's something that feels cheaper but with the pace of things, unless you are churning an insane amount of tokens, probably doesn't make sense. Never mind that local models running on 24 or 48GB are maybe around flash-lite in ability while being slower than SOTA models. Local models are mostly for hobby and privacy, not really efficiency.
FirmwareBurner 1y ago
I found Gemini now terrible for coding. I gave it my code blocks and told it what to change and it added tonnes and tonnes of needles extra code plus endless comments. It turned a tight code into a Papyrus. ChatGPT is better but tends to be too agreeable, never trying to disagree with what you say even if it's stupid so you end up shooting yourself in the foot. Claude seems like the best compromise. Just my two kopecks.
willseth 1y ago
Same experience here. I even built a Gem with am elaborate prompt instructing it how to be concise, but it still gives annoying long-winded responses and frequently expands the scope of its answer far beyond the prompt.
- theturtletalks 1y ago
  I feel like this is part of the AI playbook now. Launch a really strong, capable model (expensive price inference) and once users think it’s SOTA, neuter it so the cost is cheaper and most users won’t notice. The same happened with GPT-3.5. It was so good early on and got worse as OpenAI began to cut costs. I feel like when GPT-4.1 was cloaked as Optimus on Openrouter, it was really good, but once it launched, it also got worse.
  - carlos22 1y ago
    That is the capitalism' playbook all along. Its just much faster because its just software. But they do it for everything all the time.
    - andybak 1y ago
      Do you mind explaining how you see this working as a nefarious plot? I don't see an upside in this case so I'm going with the old "never ascribe to malice" etc
    - theturtletalks 1y ago
      I disagree with the comparison between LLM behavior and traditional software getting worse. When regular software declines in quality, it’s usually noticeable through UI changes, release notes, or other signals. Companies often don’t bother hiding it, since their users are typically locked into their ecosystem. LLMs, on the other hand, operate under different incentives. It’s in a company’s best interest to initially release the strongest model, top the benchmarks, and then quietly degrade performance over time. Unlike traditional software, LLMs have low switching costs, users can easily jump to a better alternative. That makes it more tempting for companies to conceal model downgrades to prevent user churn.
unshavedyak 1y ago
Yea, i had similar experiences. At first it felt like it solved complex problems really well, but then i realized i was having trouble steering it for simple things. It was also very verbose. Overall though my primary concern is the UX, and Claude Code is the UX of choice for me currently.
- sagarpatil 1y ago
  Check out zen MCP server https://github.com/BeehiveInnovations/zen-mcp-server https://github.com/BeehiveInnovations/zen-mcp-server Lets you use Gemini and OpenAI models in Claude Code.
  - cap11235 1y ago
    Ooh this seems nice. Most similar solutions monkeypatch the npm package, which is a bit icky
huevosabio 1y ago
They made it talk like buzzfeed articles for every single interaction. It's absolutely horrible
therealmarv 1y ago
not according to Aider leaderboard https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ I use only the APIs directly with Aider (so no experience with AI Studio). My feeling with Claude is that they still perform good with weak prompts, the "taste" is maybe a little better when the direction is kinda unknown by the prompter. When the direction is known I see Gemini 2.5 Pro (with thinking) on top of Claude with code which does not break. And with o4-mini and o3 I see more "smart" thinking (as if there is a little bit of brain inside these models) at the expense of producing unstable code (Gemini produces more stable code). I see problems with Claude when complexity increases and I would put it behind Gemini and o3 in my personal ranking. So far I had no reason to go back to Claude since o3-mini was released.
- stavros 1y ago
  I just spent $35 for Opus to solve a problem with a hardware side-project (I'm turning an old rotary phone into a meeting handset so I can quit meetings by hanging up, if you must know). It didn't solve the problem, it churned and churned and spent a ton of money. I was much more satisfied with o3 and Aider, I haven't tried them on this specific problem but I did quite a bit of work on the same project with them last night. I think I'm being a bit unfair, because what Claude got stuck on seems to be a hard problem, but I don't like how they'll happily consume all my money trying the same things over and over, and never say "yeah I give up".
  - alecco 1y ago
    Give them feedback.
    - stavros 1y ago
      Feedback on what?
      - CamperBob2 1y ago
        When I obtain results from one paid model that are significantly better than what I previously got from another paid model, I'll typically give a thumbs-down to the latter and point out in the comment that it was beaten by a competitor. Can't hurt.
dr_kiszonka 1y ago
They nerfed Pro 2.5 significantly in the last few months. Early this year, I had genuinely insightful conversations with Gemini 2.5 Pro. Now they are mostly frustrating. I also have a personal conspiracy theory, i.e., that once a user exceeds a certain use threshold of 2.5 Pro in the Google Gemini app, they start serving a quantized version. Of course, I have no proof, but it certainly feels that way.
- esafak 1y ago
  I wonder how smart they are about quantizing. Do they look at feedback to decide which users won't mind?
- conradkay 1y ago
  Maybe they've been focusing so much on improving coding performance with RL for the new versions/previews that other areas degraded in performance
  - dr_kiszonka 1y ago
    I think you are right and this is probably the case. Although, given that I rapidly went from +4 to 0 karma, a few other comments in this topic are grey, and at least one is missing, I am getting suspicious. (Or maybe it is just lunch time in MTV.)
- cma 1y ago
  One of the early updates improved agentic coding scores while lowering other general benchmark scores, which may have impacted those kind of conversations.
- SirensOfTitan 1y ago
  There was a significant nerf of Gemini 3-25 a little while ago, so much so that I detected it without knowing there was even a new release. Totally convinced they quantized the model quietly and improved on the coding benchmark to hide that fact. I’m frankly quite tired of LLM providers changing the model I’m paying for access to behind the scenes, often without informing me, and in Gemini’s case on the API too—at least last time I checked they updated the 3-25 checkpoint to the May update.
jasonjmcghee 1y ago
I have no inside information but feels like they quantized it. I've seen patterns that I usually only see in quantized models like getting stuck repeating a single character indefinitely
noisy_boy 1y ago
They should just roll back to the preview versions. Those were so much more even keeled and actually did some useful pushback instead of this cheerleader-on-steroids version they GA'd.
- k8sToGo 1y ago
  But they claim it's the same model and version?
  - noisy_boy 1y ago
    I don't know but it sure doesn't feel the same. I have been using Gemini 2.5 pro (preview and now GA) for a while. The difference in tone is palpable. I also noticed that the preview took longer time and the GA is faster so it could be quantization. Maybe a bunch of people with authority to decide thought that it was too slow/expensive/boring and screwed up a nice thing.
- samvher 1y ago
  Yes I was very surprised after the whole "scandal" around ChatGPT becoming too sycophantic that there was this massive change in tone from the last preview model (05-06) to the 06-05/GA model. The tone is really off-putting, I really liked how the preview versions felt like intelligent conversation partners and recognize what you're saying about useful pushback - it was my favorite set of models (the few preview iterations before this one) and I'm sad to see them disappearing. Many people on the Google AI Developer forums have also noted either bugs or just performance regression in the final model.
DangitBobby 1y ago
Same for me. I've been using Gemini 2.5 Pro for the past week or so because people said Gemini is the best for coding! Not at all my experience with Gemini 2.5 Pro, on top of being slow and flaky, the responses are kind of bad. Claud Sonnet 4 is much better IMO.
r0fl 1y ago
The context window on ai studio feels endless. All other ai’s seem to give me errors when working with large bodies of code.

chrismustcode 1y ago

When I ask it do to do something in cursor it goes full sherlock thinking about every possible outcome. Just claude 4 sonnet with thinking just has a bit think then does it

UncleOxidant 1y ago

Used to be able to use Gemini Pro free in cline. Now the API limits are so low that you immediately get messages about needing to top up your wallet and API queries just don't go through. Back to using DeepSeek R1 free in cline (though even that eventually stops after a few hours and you have to wait until the next day for it to work again). Starting to look like I need to setup a local LLM for coding - which means it's time to seriously upgrade my PC (well, it's been about 10 years so it was getting to be time anyway)

Workaccount2 1y ago
By the time you breakeven on whatever you spend on a decent LLM capable build, your hardware will be too far behind to run whatever is best locally then. It's something that feels cheaper but with the pace of things, unless you are churning an insane amount of tokens, probably doesn't make sense. Never mind that local models running on 24 or 48GB are maybe around flash-lite in ability while being slower than SOTA models. Local models are mostly for hobby and privacy, not really efficiency.

FirmwareBurner 1y ago

I found Gemini now terrible for coding. I gave it my code blocks and told it what to change and it added tonnes and tonnes of needles extra code plus endless comments. It turned a tight code into a Papyrus. ChatGPT is better but tends to be too agreeable, never trying to disagree with what you say even if it's stupid so you end up shooting yourself in the foot. Claude seems like the best compromise. Just my two kopecks.

willseth 1y ago

Same experience here. I even built a Gem with am elaborate prompt instructing it how to be concise, but it still gives annoying long-winded responses and frequently expands the scope of its answer far beyond the prompt.

theturtletalks 1y ago
I feel like this is part of the AI playbook now. Launch a really strong, capable model (expensive price inference) and once users think it’s SOTA, neuter it so the cost is cheaper and most users won’t notice. The same happened with GPT-3.5. It was so good early on and got worse as OpenAI began to cut costs. I feel like when GPT-4.1 was cloaked as Optimus on Openrouter, it was really good, but once it launched, it also got worse.
- carlos22 1y ago
  That is the capitalism' playbook all along. Its just much faster because its just software. But they do it for everything all the time.
  - andybak 1y ago
    Do you mind explaining how you see this working as a nefarious plot? I don't see an upside in this case so I'm going with the old "never ascribe to malice" etc
  - theturtletalks 1y ago
    I disagree with the comparison between LLM behavior and traditional software getting worse. When regular software declines in quality, it’s usually noticeable through UI changes, release notes, or other signals. Companies often don’t bother hiding it, since their users are typically locked into their ecosystem. LLMs, on the other hand, operate under different incentives. It’s in a company’s best interest to initially release the strongest model, top the benchmarks, and then quietly degrade performance over time. Unlike traditional software, LLMs have low switching costs, users can easily jump to a better alternative. That makes it more tempting for companies to conceal model downgrades to prevent user churn.

unshavedyak 1y ago

Yea, i had similar experiences. At first it felt like it solved complex problems really well, but then i realized i was having trouble steering it for simple things. It was also very verbose. Overall though my primary concern is the UX, and Claude Code is the UX of choice for me currently.

sagarpatil 1y ago
Check out zen MCP server https://github.com/BeehiveInnovations/zen-mcp-server https://github.com/BeehiveInnovations/zen-mcp-server Lets you use Gemini and OpenAI models in Claude Code.
- cap11235 1y ago
  Ooh this seems nice. Most similar solutions monkeypatch the npm package, which is a bit icky

huevosabio 1y ago

They made it talk like buzzfeed articles for every single interaction. It's absolutely horrible

therealmarv 1y ago

not according to Aider leaderboard https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ I use only the APIs directly with Aider (so no experience with AI Studio). My feeling with Claude is that they still perform good with weak prompts, the "taste" is maybe a little better when the direction is kinda unknown by the prompter. When the direction is known I see Gemini 2.5 Pro (with thinking) on top of Claude with code which does not break. And with o4-mini and o3 I see more "smart" thinking (as if there is a little bit of brain inside these models) at the expense of producing unstable code (Gemini produces more stable code). I see problems with Claude when complexity increases and I would put it behind Gemini and o3 in my personal ranking. So far I had no reason to go back to Claude since o3-mini was released.

stavros 1y ago
I just spent $35 for Opus to solve a problem with a hardware side-project (I'm turning an old rotary phone into a meeting handset so I can quit meetings by hanging up, if you must know). It didn't solve the problem, it churned and churned and spent a ton of money. I was much more satisfied with o3 and Aider, I haven't tried them on this specific problem but I did quite a bit of work on the same project with them last night. I think I'm being a bit unfair, because what Claude got stuck on seems to be a hard problem, but I don't like how they'll happily consume all my money trying the same things over and over, and never say "yeah I give up".
- alecco 1y ago
  Give them feedback.
  - stavros 1y ago
    Feedback on what?
    - CamperBob2 1y ago
      When I obtain results from one paid model that are significantly better than what I previously got from another paid model, I'll typically give a thumbs-down to the latter and point out in the comment that it was beaten by a competitor. Can't hurt.

dr_kiszonka 1y ago

They nerfed Pro 2.5 significantly in the last few months. Early this year, I had genuinely insightful conversations with Gemini 2.5 Pro. Now they are mostly frustrating. I also have a personal conspiracy theory, i.e., that once a user exceeds a certain use threshold of 2.5 Pro in the Google Gemini app, they start serving a quantized version. Of course, I have no proof, but it certainly feels that way.

esafak 1y ago
I wonder how smart they are about quantizing. Do they look at feedback to decide which users won't mind?
conradkay 1y ago
Maybe they've been focusing so much on improving coding performance with RL for the new versions/previews that other areas degraded in performance
- dr_kiszonka 1y ago
  I think you are right and this is probably the case. Although, given that I rapidly went from +4 to 0 karma, a few other comments in this topic are grey, and at least one is missing, I am getting suspicious. (Or maybe it is just lunch time in MTV.)
cma 1y ago
One of the early updates improved agentic coding scores while lowering other general benchmark scores, which may have impacted those kind of conversations.
SirensOfTitan 1y ago
There was a significant nerf of Gemini 3-25 a little while ago, so much so that I detected it without knowing there was even a new release. Totally convinced they quantized the model quietly and improved on the coding benchmark to hide that fact. I’m frankly quite tired of LLM providers changing the model I’m paying for access to behind the scenes, often without informing me, and in Gemini’s case on the API too—at least last time I checked they updated the 3-25 checkpoint to the May update.

jasonjmcghee 1y ago

I have no inside information but feels like they quantized it. I've seen patterns that I usually only see in quantized models like getting stuck repeating a single character indefinitely

noisy_boy 1y ago

They should just roll back to the preview versions. Those were so much more even keeled and actually did some useful pushback instead of this cheerleader-on-steroids version they GA'd.

k8sToGo 1y ago
But they claim it's the same model and version?
- noisy_boy 1y ago
  I don't know but it sure doesn't feel the same. I have been using Gemini 2.5 pro (preview and now GA) for a while. The difference in tone is palpable. I also noticed that the preview took longer time and the GA is faster so it could be quantization. Maybe a bunch of people with authority to decide thought that it was too slow/expensive/boring and screwed up a nice thing.
samvher 1y ago
Yes I was very surprised after the whole "scandal" around ChatGPT becoming too sycophantic that there was this massive change in tone from the last preview model (05-06) to the 06-05/GA model. The tone is really off-putting, I really liked how the preview versions felt like intelligent conversation partners and recognize what you're saying about useful pushback - it was my favorite set of models (the few preview iterations before this one) and I'm sad to see them disappearing. Many people on the Google AI Developer forums have also noted either bugs or just performance regression in the final model.

DangitBobby 1y ago

Same for me. I've been using Gemini 2.5 Pro for the past week or so because people said Gemini is the best for coding! Not at all my experience with Gemini 2.5 Pro, on top of being slow and flaky, the responses are kind of bad. Claud Sonnet 4 is much better IMO.

r0fl 1y ago

The context window on ai studio feels endless. All other ai’s seem to give me errors when working with large bodies of code.