4 ms·

It's also the only model that generates accurate translation and localization. No other frontier model comes close. Although Gemini's coding capabilities are su

It's also the only model that generates accurate translation and localization. No other frontier model comes close. Although Gemini's coding capabilities are subpar, its natural language processing is top-tier.

thisgoodlife 19d ago
I’m curious how you guys keep track of each model’s coding capabilities. The landscape keeps changing. I don’t suppose you benchmark all frontier models every other month, right?
- riddlemethat 19d ago
  I use them. Daily. Gemini hasn’t been a contender by comparison for a long time.
  - baq 19d ago
    True but it has a niche in SQL reviews for me. Looks like Google has a lot of good sql in their corpus and in their RL digital lobotomy factory.
  - safog 19d ago
    I wonder if it's a harness thing or a model thing at this point. I feel all coding models are quite capable for most tasks I want them to do. Most of the time I don't need what the bench tests and I'm not really giving them completely ambiguous tasks without any refinement. I only find marginal differences between models at this point and it almost feels like personality quirks in each model than anything.
    - BenzeneDream 19d ago
      When comparing OpenAI and Claude thats pretty much true, but not Gemini... And have you tried Antigravity? Yikes
      - vrosas 19d ago
        The CLI version of agy is great. Have you tried it?
        taylorfinley 19d ago
        Do you dangerously allow permissions? I absolutely cannot use it until they ship an auto approver. As it is now I have it write one bash/python script to do everything it wants to, then I review that. Otherwise it is COMPLETELY unusable and it shocks me when I hear people are using it.
        xnx 19d ago
        Sounds like they shipped some changes today that might reduce approvals: https://x.com/antigravity/status/2100001904969297980 https://x.com/antigravity/status/2100001904969297980
        andai 19d ago
        alias agy="agy --dangerously-skip-permissions"
        cute_boi 19d ago
        is anti gravity open sourced just like codex or grok code?
        vrosas 18d ago
        Yes, but I dangerously allow claude and codex, too...
        akho 18d ago
        Why not allow everything? It's not like it can do much inside the container.
        VectorLock 19d ago
        Compared to gemini-cli that they took out behind the woodshed, I hate it.
      - 19d ago
  - le-mark 19d ago
    I did a test involving implementing cobol control flow in Java for a source to source translation project. Gemini was the only model to get the edge cases. Cobol is very peculiar in this regard.
    - robotmay 19d ago
      It's very good at Elixir in my experience too. And it just does what I ask and doesn't wind me up like Opus. I don't think I've had to insult it more than once per day.
  - adventured 19d ago
    I had a typical $20 Gemini plan that I just downgraded to their $5 plan (to keep access to some of the models). It had been so long since I let Gemini work on (or review) any code / design / html (anything) that I couldn't justify bothering to keep wasting money on it. It fell behind badly over the past year. Astra might as well be an alien super intelligence at code compared to Gemini. I enjoy talking to Gemini, it is very good at conversation, I get solid answers to everyday questions. I intend to keep the $5 plan indefinitely for basic use. I don't expect they'll ever resurface as a competitor in coding with Astra & Fable et al.
- MoreThanMe 18d ago
  [flagged]
jakderrida 19d ago
I found that it's shockingly good with R. (the only language I know and can correct for) I doubt they even intended it to be, but it seems like I kept going from resorting to 3.5-3.8 (over time) to realizing that Claude and GPT, while great at Python, will make rudimentary mistakes with R; even when they compose giant complicated R code.
- sosrobahu 19d ago
  I'm guessing this is partly because of Gemini's world knowledge. I tried asking the model multiple internet humor and memes and it answered correctly around 80% of the time
deleted 19d ago
[deleted]
staticman2 19d ago
That might depend on whether you are translating fiction or nonfiction. Anecdotally I'd rate Gemini behind Claude and OpenAI models at fiction and I can't find any benchmarks showing Gemini is the clear winner at this task.

thisgoodlife 19d ago

I’m curious how you guys keep track of each model’s coding capabilities. The landscape keeps changing. I don’t suppose you benchmark all frontier models every other month, right?

riddlemethat 19d ago
I use them. Daily. Gemini hasn’t been a contender by comparison for a long time.
- baq 19d ago
  True but it has a niche in SQL reviews for me. Looks like Google has a lot of good sql in their corpus and in their RL digital lobotomy factory.
- safog 19d ago
  I wonder if it's a harness thing or a model thing at this point. I feel all coding models are quite capable for most tasks I want them to do. Most of the time I don't need what the bench tests and I'm not really giving them completely ambiguous tasks without any refinement. I only find marginal differences between models at this point and it almost feels like personality quirks in each model than anything.
  - BenzeneDream 19d ago
    When comparing OpenAI and Claude thats pretty much true, but not Gemini... And have you tried Antigravity? Yikes
    - vrosas 19d ago
      The CLI version of agy is great. Have you tried it?
      - taylorfinley 19d ago
        Do you dangerously allow permissions? I absolutely cannot use it until they ship an auto approver. As it is now I have it write one bash/python script to do everything it wants to, then I review that. Otherwise it is COMPLETELY unusable and it shocks me when I hear people are using it.
        xnx 19d ago
        Sounds like they shipped some changes today that might reduce approvals: https://x.com/antigravity/status/2100001904969297980 https://x.com/antigravity/status/2100001904969297980
        andai 19d ago
        alias agy="agy --dangerously-skip-permissions"
        cute_boi 19d ago
        is anti gravity open sourced just like codex or grok code?
        vrosas 18d ago
        Yes, but I dangerously allow claude and codex, too...
        akho 18d ago
        Why not allow everything? It's not like it can do much inside the container.
      - VectorLock 19d ago
        Compared to gemini-cli that they took out behind the woodshed, I hate it.
    - 19d ago
- le-mark 19d ago
  I did a test involving implementing cobol control flow in Java for a source to source translation project. Gemini was the only model to get the edge cases. Cobol is very peculiar in this regard.
  - robotmay 19d ago
    It's very good at Elixir in my experience too. And it just does what I ask and doesn't wind me up like Opus. I don't think I've had to insult it more than once per day.
- adventured 19d ago
  I had a typical $20 Gemini plan that I just downgraded to their $5 plan (to keep access to some of the models). It had been so long since I let Gemini work on (or review) any code / design / html (anything) that I couldn't justify bothering to keep wasting money on it. It fell behind badly over the past year. Astra might as well be an alien super intelligence at code compared to Gemini. I enjoy talking to Gemini, it is very good at conversation, I get solid answers to everyday questions. I intend to keep the $5 plan indefinitely for basic use. I don't expect they'll ever resurface as a competitor in coding with Astra & Fable et al.
MoreThanMe 18d ago
[flagged]

jakderrida 19d ago

I found that it's shockingly good with R. (the only language I know and can correct for) I doubt they even intended it to be, but it seems like I kept going from resorting to 3.5-3.8 (over time) to realizing that Claude and GPT, while great at Python, will make rudimentary mistakes with R; even when they compose giant complicated R code.

sosrobahu 19d ago
I'm guessing this is partly because of Gemini's world knowledge. I tried asking the model multiple internet humor and memes and it answered correctly around 80% of the time

deleted 19d ago

[deleted]

staticman2 19d ago

That might depend on whether you are translating fiction or nonfiction. Anecdotally I'd rate Gemini behind Claude and OpenAI models at fiction and I can't find any benchmarks showing Gemini is the clear winner at this task.