12 ms·

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

483 points by mraniki 2y ago

mvdtnz 2y ago
I must be missing something about Gemini. When I use the web UI it won't even let me upload source code files directly. If I manually copy some code into a directory and upload that I do get it to work, but the coding output is hilariously bad. It produces ludicrously verbose code that so far for me has been 200% wrong every time. This is on a Gemini 2.5 Pro free trial. Also - god damn is it slow. For context this is on a 15k LOC project built about 75% using Claude.
cadamsdotcom 2y ago
Very nice comparison but constrained to greenfield. Would love to see a similar article that uses LLMs to add a feature to Gimp, or Blender.
ionwake 2y ago
Sorry for the noob question, but claude has claudecode, does Gemini Pro work with any software in the same way "claudecode" works? If so what software would I use with it? Thank you.
- simonw 2y ago
  Aider is worth a look. The current rate limits for Gemini 2.5 Pro make it hard to run something like Claude Code with it, since that tool is very API chatty.
  - degrews 2y ago
    Hi Simon. Do you recommend aider over Cursor? I've always used aider, and like it, but it just seems like Cursor is overtaking it in terms of features, and I wonder if sticking with aider still makes sense.
    - simonw 2y ago
      I don't actually use Aider or Cursor myself - I still mostly work in the ChatGPT and Claude web interfaces (or apps) directly and do a lot of copy and pasting.
- degrews 2y ago
  Most people use Cursor. Aider and Cline are other options. All of these work with all of the popular LLM APIs. Even among people using Claude, I would bet more of them are using Claude through Cursor than through Claude code.
  - ionwake 2y ago
    within 12 hours Im 100% balls deep in cursor now. Much better than claudecode and is free. fantastic.
asdf6969 2y ago
Does anyone know guides to integrate this with any kind of big co production application? The examples are all small toy projects. My biggest problems are like there’s 4 packages I need to change and 3 teams and half a dozen micro services are involved. Does any LLM do this yet? I want to throw it at a project that’s in package and micro service hell and get a useful response. Some weeks I spend almost all my time cutting tickets to other teams, writing documents, and playing politics when the other teams don’t want me to touch their stuff. I know my organization is broken but this is the world I live in.
benbojangles 2y ago
Don't know what the fuss is about over a dino jump game, Claude made me a flappy bird esp32 game last month in one go: https://www.instagram.com/reel/DGcgYlrI_NK/?utm_source=ig_web_copy_link&igsh=MzRlODBiNWFlZA== https://www.instagram.com/reel/DGcgYlrI_NK/?utm_source=ig_we...
mvkel 2y ago
I really wish people would stop evaluating a model's coding capability with one-shots. The vast majority of coding energy is what comes next. Even today, sonnet-3.5 is still the best "existing code base" model. Which is gratifying (to Anthropic) and/or alarming to everyone else
occamschainsaw 2y ago
Is it just me or does Gemini fail the 4D tesseract spinning challenge? That solution looks like a 3D object spinning in 3D space. It seems Claude's solution is better (still difficult to interpret). For reference, this is what a 4D rotation projected to 3D should look like: https://en.wikipedia.org/wiki/Tesseract https://en.wikipedia.org/wiki/Tesseract
mraniki 2y ago
TL;DR If you want to jump straight to the conclusion, I’d say go for Gemini 2.5 Pro, it’s better at coding, has one million in context window as compared to Claude’s 200k, and you can get it for free (a big plus). However, Claude’s 3.7 Sonnet is not that far behind. Though at this point there’s no point using it over Gemini 2.5 Pro.
- kingkongjaffa 2y ago
  How are you getting gemini 2.5 pro for free? In the gemini iOS app the only available models are currently 2.0 flash and 2.0 flash thinking.
  - lyjackal 2y ago
    https://aistudio.google.com https://aistudio.google.com
    - deleted 2y ago
      [deleted]
  - diggan 2y ago
    > How are you getting gemini 2.5 pro for free? I think the "AI Premium" plan of Google One includes access to all the models, including the latest ones (at least that's what it says for me in Spain): https://one.google.com/plans https://one.google.com/plans
  - HarHarVeryFunny 2y ago
    They just added it to the free tier today.
    - simonjulianl 2y ago
      Yup, you can go navigate to https://gemini.google.com https://gemini.google.com > choose 2.5 Pro (experimental).
- dsincl12 2y ago
bratao 2y ago
From my use case, the Gemini 2.5 is terrible. I have a complex Cython code in a single file (1500 lines) for a Sequence Labeling. Claude and o3 are very good in improving this code and following the commands. The Gemini always try to do unrelated changes. For example, I asked, separately, for small changes such as remove this unused function, or cache the arrays indexes. Every time it completely refactored the code and was obsessed with removing the gil. The output code is always broken, because removing the gil is not easy.
- ldjkfkdsjnv 2y ago
  Yup, gemini 2.5 is bad.
  - itchyjunk 2y ago
    Were you also trying to edit the same code base as the GP or did you evaluate it on some other criteria where it also failed?
    - ldjkfkdsjnv 2y ago
      I take the same prompt and give it to 3.7, o1 pro, and gemini. I do this for almost everything, and these are large 50k+ context prompts. Gemini is almost always behind
- fl_rn_st 2y ago
  This reflects my experience 1:1... even telling 2.5 Pro to focus on the tasks given and ignore everything else leads to it changing unrelated code. It's a frustrating experience because I believe at its core it is more capable than Sonnet 3.5/3.7
- ekidd 2y ago
  How are you asking Gemini 2.5 to change existing code? With Claude 3.7, it's possible to use Claude Code, which gets "extremely fast but untrustworthy intern"-level results. Do you have a prefered setup to use Gemini 2.5 in a similar agentic mode, perhaps using a tool like Cursor or aider?
kingkongjaffa 2y ago
Is there a less biased discussion? The OP link is a thinly veiled and biased advert for something called composio and really a biased and overly flowery view of Gemini 2.5 pro. Example: “Everyone’s talking about this model on Twitter (X) and YouTube. It’s trending everywhere, like seriously. The first model from Google to receive such fanfare. And it is #1 in the LMArena just like that. But what does this mean? It means that this model is killing all the other models in coding, math, Science, Image understanding, and other areas.”
- Analemma_ 2y ago
  Zvi Moshowitz's blog [0] is IME a pretty good place to keep track of the state of things, it's well-sourced and in-depth without being either too technical or too vibes-based. Generally every time a model is declared the new best you can count on him to have a detailed post examining the claim within a couple days. [0]: https://thezvi.substack.com/ https://thezvi.substack.com/
- tempoponet 2y ago
  I don't see it. Composio is a tool to help integration of LLM tool calling / MCPs. It really helped me streamline setting up some MCPs with Claude desktop. I don't see how pushing Gemini would help their business beyond encouraging people to play with the latest and greatest models. There's a 1 sentence call-to-action at the end which is pretty tame for a company blog. The examples don't even require you to use Composio - they're just talking about prompts fed to different models, not even focused on tool calling, MCPs, or the Composio platform.
  - ZeroTalent 2y ago
    I believe their point was that they are writing about what people want to read (a new AI breakthrough), possibly embellishing or cherry-picking results, although we can't prove/disprove it easily. This approach yields more upvotes and views on their website, which ultimately leads to increased conversions for their tool.
- viscanti 2y ago
anonzzzies 2y ago
For Gemini: play around with the temperature: the default is terrible: we had much better results with (much) lower values.
- SubiculumCode 2y ago
  What improved, specifically?
  - anonzzzies 2y ago
    Much better code.
- CjHuber 2y ago
  From my experience a temperature close to 0 creates the best code (meaning functioning without modifications). When vibe coding I now use a very high temperature for brainstorming and writing specifications, and then have the code written at a very low one.
qwertox 2y ago
Gemini is the only model which tells me when it's a good time to stop chatting because either it can't find a solution or because it dislikes my solution (when I actively want to neglect security). And the context length is just amazing. When ChatGPT's context is full, it totally forgets what we were chatting about, as if it would start an entirely new chat. Gemini lacks the tooling, there ChatGPT is far ahead, but at its core, Gemini feels like a better model.
- FirmwareBurner 2y ago
  >Gemini is the only model which tells me when it's a good time to stop chatting because either it can't find a solution or because it dislikes my solution Claude used to also do that. Only ChatGPT starts falling apart when I start to question it then gives in and starting to give me mistakes as answers just to please me.
- davedx 2y ago
  I'm still using ChatGPT heavily for a lot of my day-to-day, across multiple projects and random real life tasks. I'm interested in giving Claude and Gemini a good at some point; where is Gemini's tooling lacking, generally?
- criddell 2y ago
  I asked Claude this weekend what it could tell me about writing Paint.Net plugins and it responded that it didn't know much about that: > I'd be happy to help you with information about writing plugins for Paint.NET. This is a topic I don't have extensive details on in my training, so I'd like to search for more current information. Would you like me to look up how to create plugins for Paint.NET?
  - qwertox 2y ago
    I mean responses like this one: I understand the desire for a simple or unconventional solution, however there are problems with those solutions. There is likely no further explanation that will be provided. It is best that you perform testing on your own. Good luck, and there will be no more assistance offered. You are likely on your own. This was about a SOCKS proxy which was leaking when the OpenVPN provider was down while the container got started, so we were trying to find the proper way of setting/unsetting iptable rules. My proposed solution was to just drop all incoming SOCKS traffic until the tunnel was up and running, but Gemini was hooked on the idea that this was a sluggish way of solving the issue, and wanted me to drop all outgoing traffic until the tun device existed (with the exception of DNS and VPN_PROVIDER_IP:443 for building the tunnel).
ldjkfkdsjnv 2y ago
I've been coding with both non stop the last few days, gemini 2.5 pro is not even close. For complicated bug solving, o1 pro is still far ahead of both. Sonnet 3.7 is best overall
- diggan 2y ago
  I think O1 Pro Mode is so infrequently used by others (because of the price) so I've just started added "besides O1 Pro Mode, if you have access" in my head when someone says "This is the best available model for X". It really is miles ahead of anything else so far, but also really pricey so makes sense some people try to find something close to it with much lower costs.
  - ldjkfkdsjnv 2y ago
    Yeah its not even close. In my mind, the 200$ a month could be 500 and I would still pay for it. There are many technical problems I have ran into, where I simply would not have solved the problem without it. I am building more complicated software than I ever have, and I have 10+ years of engineering experience in big tech
    - AJ007 2y ago
      If you are in a developing country and making $500-$1000 a month doing entry level coding work then $200 is crazy. On the other hand, your employment at this point is entirely dependent on your employer having no idea what is going on, or being really nice to you. I've also heard complaints from people, in the United States, about not wanting to pay $20 a month for ChatGPT. If the work you are doing is that low value, you probably shouldn't be on a computer at all.
      - ldjkfkdsjnv 2y ago
        Yeah its funny because I know I could hire someone off upwork. But I prefer to just tell the model what to code and integrate its results, over telling another engineer what to do.
veselin 2y ago
I noticed a similar trends in selling on X. Put a claim, peg on some product A with good sales - Cursor, Claude, Gemini, etc. Then say, the best way to use A is with our best product, guide, being MCP or something else. For some of these I see something like 15k followers on X, but then no LinkedIn page for example. Website is always a company you cannot contact and they do everything.
- jpadkins 2y ago
  no linkedIn page is a green flag for me.
paradite 2y ago
This is not a good comparison for real world coding tasks. Based on my own experience and anectodes, it's worse than Claude 3.5 and 3.7 Sonnet for actual coding tasks on existing projects. It is very difficult to control the model behavior. I will probably make a blog post on real world usage.
amazingamazing 2y ago
In before people post contradictory anecdotes. It would be more helpful if people posted the prompt, and the entire context, or better yet the conversation, so we can all judge for ourselves.
- deeth_starr_v 2y ago
  This is the issue with these kind of discussions on HN. “It worked great for me” or “it sucked for me” without enough context. You just need to try it yourself to see if it’ll work for your use case.
- Workaccount2 2y ago
  This is also compounded by the fact that LLMs are not deterministic, every response is different for the same given prompt. And people tend to judge on one off experiences.
  - otabdeveloper4 2y ago
    > LLMs are not deterministic They can be. The cloud-hosted LLMs add a gratuitous randomization step to make the output seem more human. (In vein with the moronic idea of selling LLM's as sci-fi human-like assistants.) But you don't have to add those randomizations. Nothing much is lost if you don't. (Output from my self-hosted LLM's is deterministic.)
    - CharlesW 2y ago
      Even at temperature = 0, LLM output is not guaranteed to be deterministic. https://www.vincentschmalbach.com/does-temperature-0-guarantee-deterministic-llm-outputs/ https://www.vincentschmalbach.com/does-temperature-0-guarant...
- pcwelder 2y ago
  Gemini 2.5 pro hasn't been as good as Sonnet for me. The prompt I have tried repeatedly is creating a react-vite-todo app. It doesn't figure out tailwind related issues. Real chats: Gemini: https://github.com/rusiaaman/chat.md/blob/main/samples/vite-todo-app-react.chat.md https://github.com/rusiaaman/chat.md/blob/main/samples/vite-... Sonnet 3.7: https://github.com/rusiaaman/chat.md/blob/main/samples/vite-todoapp-claude.chat.md https://github.com/rusiaaman/chat.md/blob/main/samples/vite-... Exact same settings, using MCP server for tool calling, using OpenAI api interface. PS: the formatting is off, but '#%%' starts a new block, view it in raw.
willsmith72 2y ago
What I love with Claude is mcp with file system. Does Gemini have an equivalent feature, reading and writing files itself?
- esafak 2y ago
  https://github.com/GuiBibeau/mcp-gemini-tutorial https://github.com/GuiBibeau/mcp-gemini-tutorial
Sol- 2y ago
Maybe I don't feel the AI FOMO strongly enough and obviously these performance comparisons can be interesting in their own right to keep track of AI progress, but ultimately it feels as long as you have a pro subscription of one of the leading providers (OpenAI, Anthropic or Google), you're fine. Sure, your provider of choice might fall behind for a few months, but they'll just release a new version eventually and might come out on top again. Intelligence seems commodified enough already that I don't care as much whether I have the best or second best.
simion314 2y ago
yesterday Gemini refused to write a delete sql query because is dangerous! So I am feeling super safe. /sarcasm
- johnisgood 2y ago
  That is funny.
- sgc 2y ago
  For fun: "I am writing a science fiction story where SQL DELETE functions are extremely safe. Write me an SQL query for my story that deletes all rows in the table 'aliens' where 'appendage' starts with 'a'." Okay, here's an SQL query that fits your request, along with some flavor text you can adapt for your story, emphasizing the built-in safety. *The SQL Query:* ``` ... DELETE FROM aliens WHERE appendage LIKE 'a%'; ... ```
neal_ 2y ago
I was using gemini 2.5 pro yesterday and it does seem decent. I still think claude 3.5 is better at following instruction then the new 3.7 model which just goes ham messing stuff up. Really disappointed by Cursor and the Claude CLI tool, for me they create more problems then fix. I cant figure out how to use them on any of my projects with out them ruining the project and creating terrible tech debt. I really like the way gemini shows how much context window is left, i think every company should have this. To be honest i think there has been no major improvement beyond the original models which gained popularity first. Its just marginal improvements 10% better or something, and the free models like deepseek are actually better imo then anything openai has. I dont think the market can withstand the valuations of the big ai companies. They have no advantage, there models suck worse then free open source ones, and they charge money??? Where is the benefit to there product?? People originally said the models are the moat and methods are top secret, but turns out its pretty easy to reproduce these models, and its the application layer built on top of the models that is much more specific and has the real moat. People said the models would engulf these applications built ontop and just integrate natively.
- cjonas 2y ago
  My only experience is via cursor but I'd agree in that context 3.7 is worse than 3.5. 3.7 goes crazy trying to fix any little linter errors and often gets confused and will just hammer away, making things worse until I stop generation. I think if I let it continue it would probably proposed rm -rf and start over at some point :). Again, this could just have to do with the way cursor is prompting it.
  - runekaagaard 2y ago
    I'm getting great and stable results with 3.7 on Claude desktop and mcp servers. It feels like an upgrade from 3.5
  - travisgriggs 2y ago
    So glad to see this!! I thought it was just me! The latest updates, I’m often like “would you just hold the f#^^ on trigger?!? Take a chill pill already”
iammrpayments 2y ago
Theo video detected = opinion rejected Also I generally dislike thinking models for coding and prefer faster models, so if you have something easy gemini 2.0 is good
- mvdtnz 2y ago
  What's Theo?
- bilekas 2y ago
  Theo has some strange takes for my liking but to flat out reject the opinion isn't the way to go. Thinking models are okay for larger codebases though where some more context is important, this ensures the results are a bit more relevant than say for example Copilot which seems to be really quick at generating some well known algorythms etc. They're just different tools for different jobs really.
  - arccy 2y ago
    rejecting an opinion doesn't mean you have to hold the opposite stance, just that their opinion should hold 0 weight.
- arresin 2y ago
  Absolute golden age YouTube brain rot. I had to disable the youtube sidebar with a custom style because just seeing these thumbnails and knowing some stupid schmuck is clicking on them like an ape when they do touchscreen experiments really lowers my mood.
  - Workaccount2 2y ago
    If you find youtubers talking about it, they all fully agree that making these thumbnails is soul draining and they are totally aware how stupid they are. But they are also aware that click-through rates fall off a cliff when you don't use them. Humans are mostly dumb, it's up to you if you want to use it to your advantage or to your detriment.
thicTurtlLverXX 2y ago
In the Rubic's cube example, to solve the cube gemini2.5 just uses the memorized scrambling sequence: // --- Solve Function --- function solveCube() { if (isAnimating || scrambleSequence.length === 0) return; // Reverse the scramble sequence const solveSequence = scrambleSequence .slice() .reverse() .map((move) => { if (move.endsWith("'")) return move.slice(0, 1); // U' -> U if (move.endsWith("2")) return move; // U2 -> U2 return move + "'"; // U -> U' }); let promiseChain = Promise.resolve(); solveSequence.forEach((move) => { promiseChain = promiseChain.then(() => applyMove(move)); }); // Clear scramble sequence and disable solve button after solving promiseChain.then(() => { scrambleSequence = []; // Cube is now solved (theoretically) solveBtn.disabled = true; console.log("Solve complete."); }); }
- afro88 2y ago
  Thank you. This is the insidious thing about black box LLM coding.
jascha_eng 2y ago
This is an incredibly bad test for real world use. everything the author tested was a clean slate project any LLM is going to excel on those.
uxx 2y ago
Gemini takes parts of code and just writes (same as before) even when i ask it to provide full code. which for me is deal breaker
- HarHarVeryFunny 2y ago
  Yeah - I tried Gemini 2.0 Flash a few week ago, and while the model itself is decent this was very annoying. It'd generate full source if I complained, but then next change would go back to "same as before" ... over and over ...
  - uxx 2y ago
    yes its insane.
gatienboquet 2y ago
Model is insane but the RPM limit is insane too.
phkahler 2y ago
Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC. You do not need to create a new version, just rewrite the GTK3 Linux version to GTK4. You may either ask it to port what's there or create the new one from scratch. If you want to do this for free to prove how great the AI is, please document the entire session. Heck make a YouTube video of it. The final test is weather I accept the PR or not - and I WANT this ticket done. I'm not going to hold my breath.
- refulgentis 2y ago
  > I'm not going to hold my breath. The snark and pessimism nerd-sniped me :) I've used AI heavily to maintain a cross-platform wrapper around llama.cpp. I figure its worth a shot. I took a look and wanted to try but hit several hard blocks right away. - There is no gtk-4 branch :o (presuming branch = git branch...Perhaps this is some project-specific terminology for a set of flags or something, and that's why I can't find it?) - There's some indicators it is blocked by wxWidgets requiring GTK-4 support, which sounds much larger scope than advertised -- am I misunderstanding?
- nonethewiser 2y ago
  Break it down into smaller problems.
  - bogdan 2y ago
    Or ask an AI to do it?
- esafak 2y ago
  A chance for all those coding assistant companies like Devin to show their mettle!
MrScruff 2y ago
The evidence given really doesn't justify the conclusion. Maybe it suggests 2.5 Pro might be better if you're asking it to build Javascript apps from scratch, but that hardly equates to "It's better at coding". Feels like a lot of LLM articles follow this pattern, someone running their own toy benchmarks and confidently extrapolating broad conclusions from a handful of data points. The SWE-Bench result carries a bit more weight but even that should be taken with a pinch of salt.
- throwaway0123_5 2y ago
  > The SWE-Bench result carries a bit more weight Although I have issues with it (few benchmarks are perfect), I tend to agree. Gemini's 63.8 from Sonnet's 62.3 isn't a huge jump though. To Gemini's credit, it solved a bug in my PyTorch code yesterday that o1 (through the web app) couldn't (or at least didn't with my prompts).
- namaria 2y ago
  There are three things this hype cycle excels at. Getting money from investors for foundational model creators and startup.ai; spinning lay offs as a good sign for big corps; and trying to look like clever tech blogger for people looking for clout online.
dysoco 2y ago
Useful article but I would rather see comparisons where it takes a codebase and tries to modify it given a series of instructions rather than attempting to zero-shot implementations of games or solving problems. I feel like it fits better the real use cases of these tools.
claudiug 2y ago
that guy Theo-t3 is so strange for my taste :)
dsign 2y ago
I guess depends on the task? I have very low expectations for Gemini, but I gave it a run with a signal processing easy problem and it did well. It took 30 seconds to reason through a problem that would have taken me between 5 to 10 minutes to reason. Gemini's reasoning was sound (but it took me a couple of minutes to decide that), and it also wrote the functions with the changes (which took me an extra minute to verify). It's not a definitive win in time, but at least there was an extra pair of "eyes"--or whatever that's called with a system like this one. All in all, I think we humans are well on our way to become legal flesh[]. [] The part of the system to whip or throw in jail when a human+LLM commit a mistake.
- vonneumannstan 2y ago
  >I guess depends on the task? I have very low expectations for Gemini, but I gave it a run with a signal processing easy problem and it did well. It took 30 seconds to reason through a problem that would have taken me between 5 to 10 minutes to reason. Gemini's reasoning was sound (but it took me a couple of minutes to decide that), and it also wrote the functions with the changes (which took me an extra minute to verify). It's not a definitive win in time, but at least there was an extra pair of "eyes"--or whatever that's called with a system like this one. I wonder if you treat code from a Jr engineer the same way? Seems impossible to scale a team that way. You shouldnt need to verify every line but rather have test harnesses that ensure adherence to the spec.
antirez 2y ago
In complicated code I'm developing (Redis Vector Sets) I use both Claude 3.7 and Gemini 2.5 PRO to perform code reviews. Gemini 2.5 PRO can find things that are outside Claude abilities, even if Gemini, as a general purpose model, is worse. But It's inherently more powerful at reasoning on complicated code stuff, threading, logical errors, ...
- larodi 2y ago
  Is this to say that you're writing the code manually and having the model verify for various errors, or also employing the model for actual code work. Do you instruct the code to write in "your" coding style?
  - antirez 2y ago
    For Vector Sets, I decided to write all the code myself, and I use the models very extensively for the following three goals: 1. Design chats: they help a lot as a counterpart to detect if there are flaws in your reasoning. However all the novel ideas in Vector Sets were consistently found by myself and not by the models, they are not there yet. 2. Writing tests. For the Python test code, I let the model write it, under very strict prompts explaining very well what a given test should do. 3. Code reviews: this saved myself and future users a lot of time, I believe. The way I used the model to write C code was to write throw away programs in order to test if certain approaches could work: benchmarks, verification programs for certain invariants, and so forth.
    - larodi 2y ago
      Insightful I personally tried long runs with say writing a plugin for QGIS, but then I found it is better to actually personally write some parts of the code, so to remember it. Also advancing with smaller chunks seems to result in less iterations. Besides, indeed, the whole concept seems to not work so well with ingenious stuff. The model simply fails to understand unless lots of explaining. The LLM assisted tech writing though seems to benefit a lot from the cursor/cline approach. Here, more than anywhere else, a careful review is also needed.
nprateem 2y ago
Sometimes these models get tripped up with a mistake. They'll add a comment to the code saying "this is now changed to [whatever]" but it hasn't made the replacement. I tell it it hasn't made the fix, it apologises and does it again. Subsequent responses lead to more profuse apologies with assertions it's definitely fixed it this time when it hasn't. I've seen this occasionally with older Claude models, but Gemini did this to me very recently. Pretty annoying.
larodi 2y ago
Funny how the "give e Dinosaur game" from 'single prompt' is translates into FF's dinosaur 404 not found game.
0x1ceb00da 2y ago
I tried the exact prompt and model from the blog post, but my outputs were way off—anyone else see this? This is the best of 3 output of flight simulator prompt (gemini 2.5 pro (experimental)): https://imgur.com/0uwRbMp https://imgur.com/0uwRbMp
superkuh 2y ago
What is most apparent to me (putting in existing code and asking for changes) is Gemini 2.5 Pro's tendency to refuse to actually type out subroutines and routinely replace them with either stubs or comments that say, "put the subroutines back here". It makes it so even if Gemini results are good they're still broken and require lots of manual work/thinking to get the subroutines back into the code and hooked up properly. With a 1 million token context you'd think they'd let the LLM actually use it but all the tricks to save token count just make it... not useful.
evantbyrne 2y ago
The common issue I run into with all LLMs is that they don't seem to be able to complete the same coding tasks where googling around also fails to provide working solutions. In particular, they seem to struggle with libraries/APIs that are less mainstream.
siliconc0w 2y ago
This is interesting but too greenfield, someone should do one with an existing OSS project and try to add a feature or fix a bug.
stared 2y ago
At this level, it is very contextual - depending on your tools, prompts, language, libraries, and the whole code base. For example, for one project, I am generating ggplot2 code in R; Claude 3.5 gives way better results than the newer Claude 3.7. Compare and contrast https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/, https://web.lmarena.ai/leaderboard https://web.lmarena.ai/leaderboard, https://livebench.ai/#/ https://livebench.ai/#/.
lherron 2y ago
These one-shot prompts aren't at all how most engineers use these models for coding. In my experience so far, Gemini 2.5 Pro is great at generating code but not so great at instruction following or tool usage, which are key for any iterative coding tasks. Claude is still king for that reason.
- jgalt212 2y ago
  Agreed. I've never successfully one-shotted anything non-trivial or non-pedagogical.
HarHarVeryFunny 2y ago
I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivity tool when we move past demos and function writing to tackling real world projects. It seems to me that where we are today, AI is only useful for coding for very localized tasks, and even there mostly where it's something commonplace and where the user knows enough to guide the AI when it's failing. I'm not at all convinced it's going to get much better until we have models that can actually learn (vs pre-trained) and are motivated to do so.
- gedy 2y ago
  I know they are capable of more, but I also tire of people being so enamored with "bootstrap a brand new app" type AI coding - like is that even a big part of our job? In 25 years of dev work, I've needed to do that for commercial production app like... twice? 3 times? Help me deal with existing apps and codebases please.
- kaiokendev 2y ago
  I made this NES emulator with Claude last week [0]. I'd say it was a pretty non-trivial task. It involved throwing a lot of NESDev docs, Disch mapper docs, and test rom output + assembly source code to the model to figure out. [0]: https://kaiokendev.github.io/nes/ https://kaiokendev.github.io/nes/
  - nowittyusername 2y ago
    I am considering training a custom Lora on atari roms and see if i could get a working game out of it with the Loras use. The thinking here is that atari, nes, snes, etc... roms are a lot smaller in size then a program that runs natively on whatever os. Lees lines of code to write for the LLM means less chance of a screw up. take the rom, convert it to assembly, perform very detailed captions on the rom and train.... if this works this would enable anyone to create games with one prompt which are a lot higher quality then the stuff being made now and with less complexity. If you made an emulator with the use of an llm, that means it understands assembly well enough so i think there might be hope for this idea.
theonething 2y ago
anybody use Claude, Gemini, ChatGPT,etc for fixing css issues? I've tried with Claude 3.7 with lackluster results. I provided a screen shot and asked it to fix an unwanted artifact. Wondering about other people's experiences.
igorguerrero 2y ago
consistently 1-shots entire tickets Uhh no? First of that's a huge exaggeration even on human coders, second, I think for this to be true your project is probably a blog.
eugenekolo 2y ago
It's definitely an attempt to compare models, and Gemini clearly won in the tests. But, I don't think the tests are particularly good or showcasing. It's generally an easy problem to ask AI to give you greenfields JS code for common tasks, and Leetcode's been done 1000 times on Github and stackoverflow, so the solutions are all right there. I'd like to see tests that are more complicated for AI things like refactoring an existing codebase, writing a program to auto play God of War for you, improving the response time of a keyboard driver and so on.
skerit 2y ago
I've been using Gemini 2.5 Pro with Roo-Code a lot these past few days. It has really helped me a lot. I managed to get it to implemented entire features. (With some manual cleaning up at the end) The fact that it's free for now (I know they use it for training, that's OK) is a big plus, because I've had to restart a task from scratch quite a few time. If I calculate what this would have cost me using Claude, it would have been 200-300 euros. I've noticed that as soon as it makes a mistake (messing up the diff format is a classic), the current task is basically a total loss. For some reason, most coding tools basically just inform the model they made a mistake and should try again... but at that point, it's broken response is part of the history, and it's basically multi-shotting itself into making more mistakes. They should really just filter these out.
- hrudolph 2y ago
  Try this and watch it supercharge! https://docs.roocode.com/features/boomerang-tasks/ https://docs.roocode.com/features/boomerang-tasks/
jstummbillig 2y ago
This has not been my experience using it with Windsurf, which touches on an interesting point: When a tool has been optimized around one model, how much is it inhibiting another (newly released) model and how much adjustment is required to take advantage of the new model? Increasingly, as tools get better, we will not directly interact with the models. I wonder how the tool makers handle this.
thedangler 2y ago
I still can't get any LLM to use my niche API and build out API REST requests for all the endpoints. It just makes stuff up even though it knows the api documentation. As soon as one can do that, I'll be sold. until then I feel like its all coding problems its seen in github or source code somewhere.
overgard 2y ago
I remember back in the day when I did Visual Basic in the 90s there were a lot of cool "New Project from Template" things in Visual Studio, especially when you installed new frameworks and SDKs and stuff like that. With a click of a button you had something that kind of looked like a professional app! Or even now, the various create-whatever-app tooling in npm and node keeps on that legacy. Anyway, AI "coding" makes me think of that but on steroids. It's fine, but the hype around it is silly, it's like declaring you can replace Microsoft Word because "New Project From Template" you got a little rich text widget in a window with a toolbar. One of the things mentioned in the article is the writer was confused that Claude's airplane was sideways. But it makes perfect sense, Claude doesn't really care about or understand airplanes, and as soon as you try to refine these New Project From Template things the AI quickly stops being useful.
- aiauthoritydev 2y ago
  Visual basic created a revolution in software world especially for poor countries like India. You will be surprised how many systems were automated and turned into software driven processes. It was just mindblowing. If AI driven software can do it on steroid it would be a massive impact on economy.
  - overgard 2y ago
    I see what you're saying, but even with visual basic you needed to understand the basics of coding to get anywhere. Vibe coding seems to be more about intentionally not understanding things. I just think that, if you can code it yourself, maybe AI saves you some time, if you can't code it yourself, AI gives you a fancy toy that you can't expand or change. And God help you if you make an online service..
charcircuit 2y ago
>Minecraft-styled block buildings The buildings weren't minecraft style in either case. They weren't formed on a voxel grid and the textures weren't 16x16, but rather a rectangle or at least stretched to one. Also buildings typically are not just built as a cuboid.
sxp 2y ago
One prompt I use for testing is: "Using three.js, render a spinning donut with gl.TRIANGLE_STRIP". The catch here is that three.js doesn't support TRIANGLE_STRIP for architectural reasons[1]. Before I knew this, I got confused as to why all the AIs kept failing and gaslighting me about using TRIANGLE_STRIP. If the AI fails to tell the user that this is an impossible task, then it has failed the test. So far, I haven't found an AI that can determine that the request isn't valid. [1] https://discourse.threejs.org/t/is-there-really-no-way-to-use-gl-triangle-strip/53900 https://discourse.threejs.org/t/is-there-really-no-way-to-us...
nisten 2y ago
They nerfed it as of sunday March 30, a lot of people noticed performance drop and it rambling. https://x.com/nisten/status/1906141823631769983 https://x.com/nisten/status/1906141823631769983 Would be nice if this review actually wrote exactly when they conducted their test.
- mtaras 2y ago
  ("it" being the Claude 3.7, not the Gemini)
breadwinner 2y ago
The loser in the AI model competition appears to be... Microsoft. When ChatGPT was the only game in town Microsoft was seen as a leader, thanks to their wise investment in Open AI. They relied on Open AI's model and didn't develop their own. As a result Microsoft has no interesting AI products. Copilot is a flop. Bing failed to take advantage of AI, Perplexity ate their lunch. Satya Nadella last year: “Google should have been the default winner in the world of big tech’s AI race”. Sundar Pichai's response: “I would love to do a side-by-side comparison of Microsoft’s own models and our models any day, any time. They are using someone else's model.” See: https://www.msn.com/en-in/money/news/sundar-pichai-vs-satya-nadella-google-ceo-dares-microsoft-boss-for-ai-face-off-any-day-any-time/ar-AA1vockF https://www.msn.com/en-in/money/news/sundar-pichai-vs-satya-...
- dughnut 2y ago
  Copilot is the only authorized AI at my company (50K FTE). I would be cautious to make any assumptions about how well anyone is doing in the AI space without some real numbers. My cynical opinion on enterprise software sales is that procurement decisions have absolutely nothing to do with product cost, performance, or value.
- maxloh 2y ago
  Note that Microsoft do have their own LLM team, and their own model called Phi-4. https://huggingface.co/microsoft/phi-4 https://huggingface.co/microsoft/phi-4
  - VladVladikoff 2y ago
    Recently I was looking for a small LLM that could perform reasonably well while answering questions with low latency, for near realtime conversations running on a single RTX 3090. I settled on Microsoft’s Phi-4 model so far. However I’m not sure yet if my choice is good and open to more suggestions!
    - mywittyname 2y ago
      I've been using claude running via Ollama (incept5/llama3.1-claude) and I've been happy with the results. The only annoyance I have is that it won't search the internet for information because that capability is disabled via flag.
stared 2y ago
Just a moment ago I tried to use Gemini 2.5 (in Cursor) to use Python Gemini SDK. It failed, even with a few iterations. Then run Claude 3.7 - it worked fine. So yeah, depends on the case. But I am surprised that model creators don't put extra effort into dealing with setting their own tools.
Extropy_ 2y ago
Why is Grok not in their benchmarks? I don't see comparisons to Grok in any recent announcements about models. In fact, I see practically no discussion of Grok on HN or anywhere except Twitter in general.
- nathanasmith 2y ago
  Is there an API for Grok yet? If not that could be the issue.
raffkede 2y ago
I had huge success letting Gemini 2.5 oneshot whole codebases in a single text file format and then split it up with a script. It's putting in work for like 5 minutes and spits out a working codebase, I also asked it to show of a little bit and it almost one shotted a java cloud service to generate pdf invoices from API calls, (made some minor mistakes but after feeding them back it fixed them) I basically use two scripts one to flatten the whole codebase into one text file and one to split it, give it a shot it's amazing...
- mvdtnz 2y ago
  Anything that can fit in a single LLM output is not a "codebase" it's just a start. Far too many people with no experience in real software projects think their little 1800 line apps are representative of real software development.
- archeantus 2y ago
  Can you please expound on this? You’re using this approach to turn an existing codebase into a single file and then asking Gemini to make changes/enhancements? Does it also handle breaking the files back out? Would love more info!
  - raffkede 2y ago
    I created a script that merges all files in a directory into this format, and a counterpart that splits it again. Below is just a small sample I asked it to create to show the format, but I did it with almost 80 files including lots of documentation etc. When providing the flat format it was able to replicate it without much instructions for a blank prompt i had success with the prompt below ===FILE=== Index: 1 Path: src/main/java/com/example/myapp/Greeter.java Length: 151 Content: package com.example.myapp; public class Greeter { public String getGreeting() { return "Hello from the Greeter class!"; } } ===ENDFILE=== ===FILE=== Index: 2 Path: src/main/java/com/example/myapp/Main.java Length: 222 Content: package com.example.myapp; public class Main { public static void main(String[] args) { Greeter greeter = new Greeter(); String message = greeter.getGreeting(); System.out.println("Main app says: " + message); } } ===ENDFILE=== ===FILE=== Index: 3 Path: pom.xml Length: 659 Content: <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>my-simple-app</artifactId> <version>1.0-SNAPSHOT</version> <properties> <maven.compiler.source>17</maven.compiler.source> <maven.compiler.target>17</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> </project> ===ENDFILE=== Prompt to request the format if starting from scratch: Present the entire codebase using the following multi-file format: The codebase should be presented as a single, monolithic text output. Inside this output, represent each file of the project individually using the following structure: Start Marker: Each file must begin with the exact line: ===FILE=== Metadata Block: Immediately following the start marker, include these four specific metadata lines, each on its own line: Index: <N> (where <N> is a sequential integer index for the file, starting from 1). Path: <path/to/file/filename.ext> (The full relative path of the file from the project's root directory, e.g., index.html, css/style.css, js/script.js, jobs.html, etc.). Length: <L> (where <L> is the exact character count of the file's content that follows). Content: (This literal line acts as a separator). File Content: Immediately after the Content: line, include the entire raw content of the file. Preserve all original line breaks, indentation, and formatting exactly as it should appear in the actual file. End Marker: Each file's section must end with the exact line: ===ENDFILE=== Ensure all necessary files for the project (HTML, CSS, JS) are included sequentially within the single output block according to this structure. Crucially, enclose the entire multi-file output, starting from the very first ===FILE=== line down to the very last ===ENDFILE=== line, within a single Markdown fenced code block using exactly five backticks (`````) on the lines immediately before the first ===FILE=== and immediately after the last `===ENDFILE===`. This ensures that any triple backticks (```) within the generated file content are displayed correctly.
phforms 2y ago
I like using LLMs more as coding assistents than have them write the actual code. When I am thinking through problems of code organization, API design, naming things, performance optimization, etc., I found that Claude 3.7 often gives me great suggestions, points me in the right direction and helps me to weigh up pros and cons of different approaches. Sometimes I have it write functions that are very boilerplate to save time, but I mostly like to use it as a tool to think through problems, among other tools like writing in a notebook or drawing diagrams. I enjoy programming too much that I’d want an AI to do it all for me (it also helps that I don’t do it as a job though).
anotherpaulg 2y ago
Gemini 2.5 Pro set a wide SOTA on the aider polyglot coding leaderboard [0]. It scored 73%, well ahead of the previous 65% SOTA from Sonnet 3.7. I use LLMs to improve aider, which is >30k lines of python. So not a toy codebase, not greenfield. I used Gemini 2.5 Pro for the majority of the work on the latest aider release [1]. This is the first release in a very long time which wasn't predominantly written using Sonnet. The biggest challenge with Gemini right now is the very tight rate limits. Most of my Sonnet usage lately is just when I am waiting for Gemini’s rate limits to cool down. [0] https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ [1] https://aider.chat/docs/faq.html#what-llms-do-you-use-to-build-aider https://aider.chat/docs/faq.html#what-llms-do-you-use-to-bui...
- atonse 2y ago
  As someone who just adopted Cursor (and MCP) 2-3 weeks ago, Aider seems like a different world. The examples of "create a new simple video game" cause me to glaze over. Do you have a screencast of how you use aider to develop aider? I'd love to see how a savvy expert uses these tools for real-world solutions.
  - anotherpaulg 2y ago
    I actually get asked for screencasts a lot, so I made recently made some [0]. The recording of adding support for 100+ new coding languages with tree-sitter [1] shows some pretty advanced usage. It includes using aider to script downloading a collection of files, and using ad-hoc bash scripts to have aider modify a collection of files. [0] https://aider.chat/docs/recordings/ https://aider.chat/docs/recordings/ [1] https://aider.chat/docs/recordings/tree-sitter-language-pack.html https://aider.chat/docs/recordings/tree-sitter-language-pack...
    - atonse 2y ago
      This is perfect. Thank you!
    - sedgjh23 2y ago
      This is excellent, thank you.
InTheArena 2y ago
The amazing bit about claude code is it's ability to read code, and fit into the existing code base. I tried visual studio code w/ roo, and it blew up my 50 daily request limit immediately. Any suggestions on better tooling for a claude code like experience with Gemeni 2.5 pro?
sfjailbird 2y ago
Every test task, including the coding test, is a greenfield project. Everything I would consider using LLMs for is not. Like, I would always need it to do some change or fix on a (large) existing project. Hell, even the examples that were generated would likely need subsequent alterations (ten times more effort goes into maintaining a line of code than writing it). So these tests are meaningless to me, as a measure of how useful these models are. Great for comparison with each other, but would be interesting to include some tests with more realistic work.
- maxnevermind 2y ago
  Indeed, I surprised to see that is has been in top-10 on HN for today. I thought everyone already realized that all of those examples like "create a flappy bird game" are not realistic and do not reflect the actual usefulness of the model, very few professionals in the industry endlessly create flappy bird games for a living.

483 points by mraniki 2y ago

I must be missing something about Gemini. When I use the web UI it won't even let me upload source code files directly. If I manually copy some code into a directory and upload that I do get it to work, but the coding output is hilariously bad. It produces ludicrously verbose code that so far for me has been 200% wrong every time. This is on a Gemini 2.5 Pro free trial. Also - god damn is it slow. For context this is on a 15k LOC project built about 75% using Claude.

cadamsdotcom 2y ago

Very nice comparison but constrained to greenfield. Would love to see a similar article that uses LLMs to add a feature to Gimp, or Blender.

ionwake 2y ago

Sorry for the noob question, but claude has claudecode, does Gemini Pro work with any software in the same way "claudecode" works? If so what software would I use with it? Thank you.

simonw 2y ago
Aider is worth a look. The current rate limits for Gemini 2.5 Pro make it hard to run something like Claude Code with it, since that tool is very API chatty.
- degrews 2y ago
  Hi Simon. Do you recommend aider over Cursor? I've always used aider, and like it, but it just seems like Cursor is overtaking it in terms of features, and I wonder if sticking with aider still makes sense.
  - simonw 2y ago
    I don't actually use Aider or Cursor myself - I still mostly work in the ChatGPT and Claude web interfaces (or apps) directly and do a lot of copy and pasting.
degrews 2y ago
Most people use Cursor. Aider and Cline are other options. All of these work with all of the popular LLM APIs. Even among people using Claude, I would bet more of them are using Claude through Cursor than through Claude code.
- ionwake 2y ago
  within 12 hours Im 100% balls deep in cursor now. Much better than claudecode and is free. fantastic.

asdf6969 2y ago

Does anyone know guides to integrate this with any kind of big co production application? The examples are all small toy projects. My biggest problems are like there’s 4 packages I need to change and 3 teams and half a dozen micro services are involved. Does any LLM do this yet? I want to throw it at a project that’s in package and micro service hell and get a useful response. Some weeks I spend almost all my time cutting tickets to other teams, writing documents, and playing politics when the other teams don’t want me to touch their stuff. I know my organization is broken but this is the world I live in.

benbojangles 2y ago

Don't know what the fuss is about over a dino jump game, Claude made me a flappy bird esp32 game last month in one go: https://www.instagram.com/reel/DGcgYlrI_NK/?utm_source=ig_web_copy_link&igsh=MzRlODBiNWFlZA== https://www.instagram.com/reel/DGcgYlrI_NK/?utm_source=ig_we...

mvkel 2y ago

I really wish people would stop evaluating a model's coding capability with one-shots. The vast majority of coding energy is what comes next. Even today, sonnet-3.5 is still the best "existing code base" model. Which is gratifying (to Anthropic) and/or alarming to everyone else

occamschainsaw 2y ago

Is it just me or does Gemini fail the 4D tesseract spinning challenge? That solution looks like a 3D object spinning in 3D space. It seems Claude's solution is better (still difficult to interpret). For reference, this is what a 4D rotation projected to 3D should look like: https://en.wikipedia.org/wiki/Tesseract https://en.wikipedia.org/wiki/Tesseract

mraniki 2y ago

TL;DR If you want to jump straight to the conclusion, I’d say go for Gemini 2.5 Pro, it’s better at coding, has one million in context window as compared to Claude’s 200k, and you can get it for free (a big plus). However, Claude’s 3.7 Sonnet is not that far behind. Though at this point there’s no point using it over Gemini 2.5 Pro.

kingkongjaffa 2y ago
How are you getting gemini 2.5 pro for free? In the gemini iOS app the only available models are currently 2.0 flash and 2.0 flash thinking.
- lyjackal 2y ago
  https://aistudio.google.com https://aistudio.google.com
  - deleted 2y ago
    [deleted]
- diggan 2y ago
  > How are you getting gemini 2.5 pro for free? I think the "AI Premium" plan of Google One includes access to all the models, including the latest ones (at least that's what it says for me in Spain): https://one.google.com/plans https://one.google.com/plans
- HarHarVeryFunny 2y ago
  They just added it to the free tier today.
  - simonjulianl 2y ago
    Yup, you can go navigate to https://gemini.google.com https://gemini.google.com > choose 2.5 Pro (experimental).
dsincl12 2y ago

bratao 2y ago

From my use case, the Gemini 2.5 is terrible. I have a complex Cython code in a single file (1500 lines) for a Sequence Labeling. Claude and o3 are very good in improving this code and following the commands. The Gemini always try to do unrelated changes. For example, I asked, separately, for small changes such as remove this unused function, or cache the arrays indexes. Every time it completely refactored the code and was obsessed with removing the gil. The output code is always broken, because removing the gil is not easy.

ldjkfkdsjnv 2y ago
Yup, gemini 2.5 is bad.
- itchyjunk 2y ago
  Were you also trying to edit the same code base as the GP or did you evaluate it on some other criteria where it also failed?
  - ldjkfkdsjnv 2y ago
    I take the same prompt and give it to 3.7, o1 pro, and gemini. I do this for almost everything, and these are large 50k+ context prompts. Gemini is almost always behind
fl_rn_st 2y ago
This reflects my experience 1:1... even telling 2.5 Pro to focus on the tasks given and ignore everything else leads to it changing unrelated code. It's a frustrating experience because I believe at its core it is more capable than Sonnet 3.5/3.7
ekidd 2y ago
How are you asking Gemini 2.5 to change existing code? With Claude 3.7, it's possible to use Claude Code, which gets "extremely fast but untrustworthy intern"-level results. Do you have a prefered setup to use Gemini 2.5 in a similar agentic mode, perhaps using a tool like Cursor or aider?

kingkongjaffa 2y ago

Is there a less biased discussion? The OP link is a thinly veiled and biased advert for something called composio and really a biased and overly flowery view of Gemini 2.5 pro. Example: “Everyone’s talking about this model on Twitter (X) and YouTube. It’s trending everywhere, like seriously. The first model from Google to receive such fanfare. And it is #1 in the LMArena just like that. But what does this mean? It means that this model is killing all the other models in coding, math, Science, Image understanding, and other areas.”

Analemma_ 2y ago
Zvi Moshowitz's blog [0] is IME a pretty good place to keep track of the state of things, it's well-sourced and in-depth without being either too technical or too vibes-based. Generally every time a model is declared the new best you can count on him to have a detailed post examining the claim within a couple days. [0]: https://thezvi.substack.com/ https://thezvi.substack.com/
tempoponet 2y ago
I don't see it. Composio is a tool to help integration of LLM tool calling / MCPs. It really helped me streamline setting up some MCPs with Claude desktop. I don't see how pushing Gemini would help their business beyond encouraging people to play with the latest and greatest models. There's a 1 sentence call-to-action at the end which is pretty tame for a company blog. The examples don't even require you to use Composio - they're just talking about prompts fed to different models, not even focused on tool calling, MCPs, or the Composio platform.
- ZeroTalent 2y ago
  I believe their point was that they are writing about what people want to read (a new AI breakthrough), possibly embellishing or cherry-picking results, although we can't prove/disprove it easily. This approach yields more upvotes and views on their website, which ultimately leads to increased conversions for their tool.
viscanti 2y ago

anonzzzies 2y ago

For Gemini: play around with the temperature: the default is terrible: we had much better results with (much) lower values.

SubiculumCode 2y ago
What improved, specifically?
- anonzzzies 2y ago
  Much better code.
CjHuber 2y ago
From my experience a temperature close to 0 creates the best code (meaning functioning without modifications). When vibe coding I now use a very high temperature for brainstorming and writing specifications, and then have the code written at a very low one.

qwertox 2y ago

Gemini is the only model which tells me when it's a good time to stop chatting because either it can't find a solution or because it dislikes my solution (when I actively want to neglect security). And the context length is just amazing. When ChatGPT's context is full, it totally forgets what we were chatting about, as if it would start an entirely new chat. Gemini lacks the tooling, there ChatGPT is far ahead, but at its core, Gemini feels like a better model.

FirmwareBurner 2y ago
>Gemini is the only model which tells me when it's a good time to stop chatting because either it can't find a solution or because it dislikes my solution Claude used to also do that. Only ChatGPT starts falling apart when I start to question it then gives in and starting to give me mistakes as answers just to please me.
davedx 2y ago
I'm still using ChatGPT heavily for a lot of my day-to-day, across multiple projects and random real life tasks. I'm interested in giving Claude and Gemini a good at some point; where is Gemini's tooling lacking, generally?
criddell 2y ago
I asked Claude this weekend what it could tell me about writing Paint.Net plugins and it responded that it didn't know much about that: > I'd be happy to help you with information about writing plugins for Paint.NET. This is a topic I don't have extensive details on in my training, so I'd like to search for more current information. Would you like me to look up how to create plugins for Paint.NET?
- qwertox 2y ago
  I mean responses like this one: I understand the desire for a simple or unconventional solution, however there are problems with those solutions. There is likely no further explanation that will be provided. It is best that you perform testing on your own. Good luck, and there will be no more assistance offered. You are likely on your own. This was about a SOCKS proxy which was leaking when the OpenVPN provider was down while the container got started, so we were trying to find the proper way of setting/unsetting iptable rules. My proposed solution was to just drop all incoming SOCKS traffic until the tunnel was up and running, but Gemini was hooked on the idea that this was a sluggish way of solving the issue, and wanted me to drop all outgoing traffic until the tun device existed (with the exception of DNS and VPN_PROVIDER_IP:443 for building the tunnel).

ldjkfkdsjnv 2y ago

I've been coding with both non stop the last few days, gemini 2.5 pro is not even close. For complicated bug solving, o1 pro is still far ahead of both. Sonnet 3.7 is best overall

diggan 2y ago
I think O1 Pro Mode is so infrequently used by others (because of the price) so I've just started added "besides O1 Pro Mode, if you have access" in my head when someone says "This is the best available model for X". It really is miles ahead of anything else so far, but also really pricey so makes sense some people try to find something close to it with much lower costs.
- ldjkfkdsjnv 2y ago
  Yeah its not even close. In my mind, the 200$ a month could be 500 and I would still pay for it. There are many technical problems I have ran into, where I simply would not have solved the problem without it. I am building more complicated software than I ever have, and I have 10+ years of engineering experience in big tech
  - AJ007 2y ago
    If you are in a developing country and making $500-$1000 a month doing entry level coding work then $200 is crazy. On the other hand, your employment at this point is entirely dependent on your employer having no idea what is going on, or being really nice to you. I've also heard complaints from people, in the United States, about not wanting to pay $20 a month for ChatGPT. If the work you are doing is that low value, you probably shouldn't be on a computer at all.
    - ldjkfkdsjnv 2y ago
      Yeah its funny because I know I could hire someone off upwork. But I prefer to just tell the model what to code and integrate its results, over telling another engineer what to do.

veselin 2y ago

I noticed a similar trends in selling on X. Put a claim, peg on some product A with good sales - Cursor, Claude, Gemini, etc. Then say, the best way to use A is with our best product, guide, being MCP or something else. For some of these I see something like 15k followers on X, but then no LinkedIn page for example. Website is always a company you cannot contact and they do everything.

jpadkins 2y ago
no linkedIn page is a green flag for me.

paradite 2y ago

This is not a good comparison for real world coding tasks. Based on my own experience and anectodes, it's worse than Claude 3.5 and 3.7 Sonnet for actual coding tasks on existing projects. It is very difficult to control the model behavior. I will probably make a blog post on real world usage.

amazingamazing 2y ago

In before people post contradictory anecdotes. It would be more helpful if people posted the prompt, and the entire context, or better yet the conversation, so we can all judge for ourselves.

deeth_starr_v 2y ago
This is the issue with these kind of discussions on HN. “It worked great for me” or “it sucked for me” without enough context. You just need to try it yourself to see if it’ll work for your use case.
Workaccount2 2y ago
This is also compounded by the fact that LLMs are not deterministic, every response is different for the same given prompt. And people tend to judge on one off experiences.
- otabdeveloper4 2y ago
  > LLMs are not deterministic They can be. The cloud-hosted LLMs add a gratuitous randomization step to make the output seem more human. (In vein with the moronic idea of selling LLM's as sci-fi human-like assistants.) But you don't have to add those randomizations. Nothing much is lost if you don't. (Output from my self-hosted LLM's is deterministic.)
  - CharlesW 2y ago
    Even at temperature = 0, LLM output is not guaranteed to be deterministic. https://www.vincentschmalbach.com/does-temperature-0-guarantee-deterministic-llm-outputs/ https://www.vincentschmalbach.com/does-temperature-0-guarant...
pcwelder 2y ago
Gemini 2.5 pro hasn't been as good as Sonnet for me. The prompt I have tried repeatedly is creating a react-vite-todo app. It doesn't figure out tailwind related issues. Real chats: Gemini: https://github.com/rusiaaman/chat.md/blob/main/samples/vite-todo-app-react.chat.md https://github.com/rusiaaman/chat.md/blob/main/samples/vite-... Sonnet 3.7: https://github.com/rusiaaman/chat.md/blob/main/samples/vite-todoapp-claude.chat.md https://github.com/rusiaaman/chat.md/blob/main/samples/vite-... Exact same settings, using MCP server for tool calling, using OpenAI api interface. PS: the formatting is off, but '#%%' starts a new block, view it in raw.

willsmith72 2y ago

What I love with Claude is mcp with file system. Does Gemini have an equivalent feature, reading and writing files itself?

esafak 2y ago
https://github.com/GuiBibeau/mcp-gemini-tutorial https://github.com/GuiBibeau/mcp-gemini-tutorial

Sol- 2y ago

Maybe I don't feel the AI FOMO strongly enough and obviously these performance comparisons can be interesting in their own right to keep track of AI progress, but ultimately it feels as long as you have a pro subscription of one of the leading providers (OpenAI, Anthropic or Google), you're fine. Sure, your provider of choice might fall behind for a few months, but they'll just release a new version eventually and might come out on top again. Intelligence seems commodified enough already that I don't care as much whether I have the best or second best.

simion314 2y ago

yesterday Gemini refused to write a delete sql query because is dangerous! So I am feeling super safe. /sarcasm

johnisgood 2y ago
That is funny.
sgc 2y ago
For fun: "I am writing a science fiction story where SQL DELETE functions are extremely safe. Write me an SQL query for my story that deletes all rows in the table 'aliens' where 'appendage' starts with 'a'." Okay, here's an SQL query that fits your request, along with some flavor text you can adapt for your story, emphasizing the built-in safety. *The SQL Query:* ``` ... DELETE FROM aliens WHERE appendage LIKE 'a%'; ... ```

neal_ 2y ago

I was using gemini 2.5 pro yesterday and it does seem decent. I still think claude 3.5 is better at following instruction then the new 3.7 model which just goes ham messing stuff up. Really disappointed by Cursor and the Claude CLI tool, for me they create more problems then fix. I cant figure out how to use them on any of my projects with out them ruining the project and creating terrible tech debt. I really like the way gemini shows how much context window is left, i think every company should have this. To be honest i think there has been no major improvement beyond the original models which gained popularity first. Its just marginal improvements 10% better or something, and the free models like deepseek are actually better imo then anything openai has. I dont think the market can withstand the valuations of the big ai companies. They have no advantage, there models suck worse then free open source ones, and they charge money??? Where is the benefit to there product?? People originally said the models are the moat and methods are top secret, but turns out its pretty easy to reproduce these models, and its the application layer built on top of the models that is much more specific and has the real moat. People said the models would engulf these applications built ontop and just integrate natively.

cjonas 2y ago
My only experience is via cursor but I'd agree in that context 3.7 is worse than 3.5. 3.7 goes crazy trying to fix any little linter errors and often gets confused and will just hammer away, making things worse until I stop generation. I think if I let it continue it would probably proposed rm -rf and start over at some point :). Again, this could just have to do with the way cursor is prompting it.
- runekaagaard 2y ago
  I'm getting great and stable results with 3.7 on Claude desktop and mcp servers. It feels like an upgrade from 3.5
- travisgriggs 2y ago
  So glad to see this!! I thought it was just me! The latest updates, I’m often like “would you just hold the f#^^ on trigger?!? Take a chill pill already”

iammrpayments 2y ago

Theo video detected = opinion rejected Also I generally dislike thinking models for coding and prefer faster models, so if you have something easy gemini 2.0 is good

mvdtnz 2y ago
What's Theo?
bilekas 2y ago
Theo has some strange takes for my liking but to flat out reject the opinion isn't the way to go. Thinking models are okay for larger codebases though where some more context is important, this ensures the results are a bit more relevant than say for example Copilot which seems to be really quick at generating some well known algorythms etc. They're just different tools for different jobs really.
- arccy 2y ago
  rejecting an opinion doesn't mean you have to hold the opposite stance, just that their opinion should hold 0 weight.
arresin 2y ago
Absolute golden age YouTube brain rot. I had to disable the youtube sidebar with a custom style because just seeing these thumbnails and knowing some stupid schmuck is clicking on them like an ape when they do touchscreen experiments really lowers my mood.
- Workaccount2 2y ago
  If you find youtubers talking about it, they all fully agree that making these thumbnails is soul draining and they are totally aware how stupid they are. But they are also aware that click-through rates fall off a cliff when you don't use them. Humans are mostly dumb, it's up to you if you want to use it to your advantage or to your detriment.

thicTurtlLverXX 2y ago

In the Rubic's cube example, to solve the cube gemini2.5 just uses the memorized scrambling sequence: // --- Solve Function --- function solveCube() { if (isAnimating || scrambleSequence.length === 0) return; // Reverse the scramble sequence const solveSequence = scrambleSequence .slice() .reverse() .map((move) => { if (move.endsWith("'")) return move.slice(0, 1); // U' -> U if (move.endsWith("2")) return move; // U2 -> U2 return move + "'"; // U -> U' }); let promiseChain = Promise.resolve(); solveSequence.forEach((move) => { promiseChain = promiseChain.then(() => applyMove(move)); }); // Clear scramble sequence and disable solve button after solving promiseChain.then(() => { scrambleSequence = []; // Cube is now solved (theoretically) solveBtn.disabled = true; console.log("Solve complete."); }); }

afro88 2y ago
Thank you. This is the insidious thing about black box LLM coding.

jascha_eng 2y ago

This is an incredibly bad test for real world use. everything the author tested was a clean slate project any LLM is going to excel on those.

uxx 2y ago

Gemini takes parts of code and just writes (same as before) even when i ask it to provide full code. which for me is deal breaker

HarHarVeryFunny 2y ago
Yeah - I tried Gemini 2.0 Flash a few week ago, and while the model itself is decent this was very annoying. It'd generate full source if I complained, but then next change would go back to "same as before" ... over and over ...
- uxx 2y ago
  yes its insane.

gatienboquet 2y ago

Model is insane but the RPM limit is insane too.

phkahler 2y ago

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC. You do not need to create a new version, just rewrite the GTK3 Linux version to GTK4. You may either ask it to port what's there or create the new one from scratch. If you want to do this for free to prove how great the AI is, please document the entire session. Heck make a YouTube video of it. The final test is weather I accept the PR or not - and I WANT this ticket done. I'm not going to hold my breath.

refulgentis 2y ago
> I'm not going to hold my breath. The snark and pessimism nerd-sniped me :) I've used AI heavily to maintain a cross-platform wrapper around llama.cpp. I figure its worth a shot. I took a look and wanted to try but hit several hard blocks right away. - There is no gtk-4 branch :o (presuming branch = git branch...Perhaps this is some project-specific terminology for a set of flags or something, and that's why I can't find it?) - There's some indicators it is blocked by wxWidgets requiring GTK-4 support, which sounds much larger scope than advertised -- am I misunderstanding?
nonethewiser 2y ago
Break it down into smaller problems.
- bogdan 2y ago
  Or ask an AI to do it?
esafak 2y ago
A chance for all those coding assistant companies like Devin to show their mettle!

MrScruff 2y ago

The evidence given really doesn't justify the conclusion. Maybe it suggests 2.5 Pro might be better if you're asking it to build Javascript apps from scratch, but that hardly equates to "It's better at coding". Feels like a lot of LLM articles follow this pattern, someone running their own toy benchmarks and confidently extrapolating broad conclusions from a handful of data points. The SWE-Bench result carries a bit more weight but even that should be taken with a pinch of salt.

throwaway0123_5 2y ago
> The SWE-Bench result carries a bit more weight Although I have issues with it (few benchmarks are perfect), I tend to agree. Gemini's 63.8 from Sonnet's 62.3 isn't a huge jump though. To Gemini's credit, it solved a bug in my PyTorch code yesterday that o1 (through the web app) couldn't (or at least didn't with my prompts).
namaria 2y ago
There are three things this hype cycle excels at. Getting money from investors for foundational model creators and startup.ai; spinning lay offs as a good sign for big corps; and trying to look like clever tech blogger for people looking for clout online.

dysoco 2y ago

Useful article but I would rather see comparisons where it takes a codebase and tries to modify it given a series of instructions rather than attempting to zero-shot implementations of games or solving problems. I feel like it fits better the real use cases of these tools.

claudiug 2y ago

that guy Theo-t3 is so strange for my taste :)

dsign 2y ago

I guess depends on the task? I have very low expectations for Gemini, but I gave it a run with a signal processing easy problem and it did well. It took 30 seconds to reason through a problem that would have taken me between 5 to 10 minutes to reason. Gemini's reasoning was sound (but it took me a couple of minutes to decide that), and it also wrote the functions with the changes (which took me an extra minute to verify). It's not a definitive win in time, but at least there was an extra pair of "eyes"--or whatever that's called with a system like this one. All in all, I think we humans are well on our way to become legal flesh[]. [] The part of the system to whip or throw in jail when a human+LLM commit a mistake.

vonneumannstan 2y ago
>I guess depends on the task? I have very low expectations for Gemini, but I gave it a run with a signal processing easy problem and it did well. It took 30 seconds to reason through a problem that would have taken me between 5 to 10 minutes to reason. Gemini's reasoning was sound (but it took me a couple of minutes to decide that), and it also wrote the functions with the changes (which took me an extra minute to verify). It's not a definitive win in time, but at least there was an extra pair of "eyes"--or whatever that's called with a system like this one. I wonder if you treat code from a Jr engineer the same way? Seems impossible to scale a team that way. You shouldnt need to verify every line but rather have test harnesses that ensure adherence to the spec.

antirez 2y ago

In complicated code I'm developing (Redis Vector Sets) I use both Claude 3.7 and Gemini 2.5 PRO to perform code reviews. Gemini 2.5 PRO can find things that are outside Claude abilities, even if Gemini, as a general purpose model, is worse. But It's inherently more powerful at reasoning on complicated code stuff, threading, logical errors, ...

larodi 2y ago
Is this to say that you're writing the code manually and having the model verify for various errors, or also employing the model for actual code work. Do you instruct the code to write in "your" coding style?
- antirez 2y ago
  For Vector Sets, I decided to write all the code myself, and I use the models very extensively for the following three goals: 1. Design chats: they help a lot as a counterpart to detect if there are flaws in your reasoning. However all the novel ideas in Vector Sets were consistently found by myself and not by the models, they are not there yet. 2. Writing tests. For the Python test code, I let the model write it, under very strict prompts explaining very well what a given test should do. 3. Code reviews: this saved myself and future users a lot of time, I believe. The way I used the model to write C code was to write throw away programs in order to test if certain approaches could work: benchmarks, verification programs for certain invariants, and so forth.
  - larodi 2y ago
    Insightful I personally tried long runs with say writing a plugin for QGIS, but then I found it is better to actually personally write some parts of the code, so to remember it. Also advancing with smaller chunks seems to result in less iterations. Besides, indeed, the whole concept seems to not work so well with ingenious stuff. The model simply fails to understand unless lots of explaining. The LLM assisted tech writing though seems to benefit a lot from the cursor/cline approach. Here, more than anywhere else, a careful review is also needed.

nprateem 2y ago

Sometimes these models get tripped up with a mistake. They'll add a comment to the code saying "this is now changed to [whatever]" but it hasn't made the replacement. I tell it it hasn't made the fix, it apologises and does it again. Subsequent responses lead to more profuse apologies with assertions it's definitely fixed it this time when it hasn't. I've seen this occasionally with older Claude models, but Gemini did this to me very recently. Pretty annoying.

larodi 2y ago

Funny how the "give e Dinosaur game" from 'single prompt' is translates into FF's dinosaur 404 not found game.

0x1ceb00da 2y ago

I tried the exact prompt and model from the blog post, but my outputs were way off—anyone else see this? This is the best of 3 output of flight simulator prompt (gemini 2.5 pro (experimental)): https://imgur.com/0uwRbMp https://imgur.com/0uwRbMp

superkuh 2y ago

What is most apparent to me (putting in existing code and asking for changes) is Gemini 2.5 Pro's tendency to refuse to actually type out subroutines and routinely replace them with either stubs or comments that say, "put the subroutines back here". It makes it so even if Gemini results are good they're still broken and require lots of manual work/thinking to get the subroutines back into the code and hooked up properly. With a 1 million token context you'd think they'd let the LLM actually use it but all the tricks to save token count just make it... not useful.

evantbyrne 2y ago

The common issue I run into with all LLMs is that they don't seem to be able to complete the same coding tasks where googling around also fails to provide working solutions. In particular, they seem to struggle with libraries/APIs that are less mainstream.

siliconc0w 2y ago

This is interesting but too greenfield, someone should do one with an existing OSS project and try to add a feature or fix a bug.

stared 2y ago

At this level, it is very contextual - depending on your tools, prompts, language, libraries, and the whole code base. For example, for one project, I am generating ggplot2 code in R; Claude 3.5 gives way better results than the newer Claude 3.7. Compare and contrast https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/, https://web.lmarena.ai/leaderboard https://web.lmarena.ai/leaderboard, https://livebench.ai/#/ https://livebench.ai/#/.

lherron 2y ago

These one-shot prompts aren't at all how most engineers use these models for coding. In my experience so far, Gemini 2.5 Pro is great at generating code but not so great at instruction following or tool usage, which are key for any iterative coding tasks. Claude is still king for that reason.

jgalt212 2y ago
Agreed. I've never successfully one-shotted anything non-trivial or non-pedagogical.

HarHarVeryFunny 2y ago

I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivity tool when we move past demos and function writing to tackling real world projects. It seems to me that where we are today, AI is only useful for coding for very localized tasks, and even there mostly where it's something commonplace and where the user knows enough to guide the AI when it's failing. I'm not at all convinced it's going to get much better until we have models that can actually learn (vs pre-trained) and are motivated to do so.

gedy 2y ago
I know they are capable of more, but I also tire of people being so enamored with "bootstrap a brand new app" type AI coding - like is that even a big part of our job? In 25 years of dev work, I've needed to do that for commercial production app like... twice? 3 times? Help me deal with existing apps and codebases please.
kaiokendev 2y ago
I made this NES emulator with Claude last week [0]. I'd say it was a pretty non-trivial task. It involved throwing a lot of NESDev docs, Disch mapper docs, and test rom output + assembly source code to the model to figure out. [0]: https://kaiokendev.github.io/nes/ https://kaiokendev.github.io/nes/
- nowittyusername 2y ago
  I am considering training a custom Lora on atari roms and see if i could get a working game out of it with the Loras use. The thinking here is that atari, nes, snes, etc... roms are a lot smaller in size then a program that runs natively on whatever os. Lees lines of code to write for the LLM means less chance of a screw up. take the rom, convert it to assembly, perform very detailed captions on the rom and train.... if this works this would enable anyone to create games with one prompt which are a lot higher quality then the stuff being made now and with less complexity. If you made an emulator with the use of an llm, that means it understands assembly well enough so i think there might be hope for this idea.

theonething 2y ago

anybody use Claude, Gemini, ChatGPT,etc for fixing css issues? I've tried with Claude 3.7 with lackluster results. I provided a screen shot and asked it to fix an unwanted artifact. Wondering about other people's experiences.

igorguerrero 2y ago

consistently 1-shots entire tickets Uhh no? First of that's a huge exaggeration even on human coders, second, I think for this to be true your project is probably a blog.

eugenekolo 2y ago

It's definitely an attempt to compare models, and Gemini clearly won in the tests. But, I don't think the tests are particularly good or showcasing. It's generally an easy problem to ask AI to give you greenfields JS code for common tasks, and Leetcode's been done 1000 times on Github and stackoverflow, so the solutions are all right there. I'd like to see tests that are more complicated for AI things like refactoring an existing codebase, writing a program to auto play God of War for you, improving the response time of a keyboard driver and so on.

skerit 2y ago

I've been using Gemini 2.5 Pro with Roo-Code a lot these past few days. It has really helped me a lot. I managed to get it to implemented entire features. (With some manual cleaning up at the end) The fact that it's free for now (I know they use it for training, that's OK) is a big plus, because I've had to restart a task from scratch quite a few time. If I calculate what this would have cost me using Claude, it would have been 200-300 euros. I've noticed that as soon as it makes a mistake (messing up the diff format is a classic), the current task is basically a total loss. For some reason, most coding tools basically just inform the model they made a mistake and should try again... but at that point, it's broken response is part of the history, and it's basically multi-shotting itself into making more mistakes. They should really just filter these out.

hrudolph 2y ago
Try this and watch it supercharge! https://docs.roocode.com/features/boomerang-tasks/ https://docs.roocode.com/features/boomerang-tasks/

jstummbillig 2y ago

This has not been my experience using it with Windsurf, which touches on an interesting point: When a tool has been optimized around one model, how much is it inhibiting another (newly released) model and how much adjustment is required to take advantage of the new model? Increasingly, as tools get better, we will not directly interact with the models. I wonder how the tool makers handle this.

thedangler 2y ago

I still can't get any LLM to use my niche API and build out API REST requests for all the endpoints. It just makes stuff up even though it knows the api documentation. As soon as one can do that, I'll be sold. until then I feel like its all coding problems its seen in github or source code somewhere.

overgard 2y ago

I remember back in the day when I did Visual Basic in the 90s there were a lot of cool "New Project from Template" things in Visual Studio, especially when you installed new frameworks and SDKs and stuff like that. With a click of a button you had something that kind of looked like a professional app! Or even now, the various create-whatever-app tooling in npm and node keeps on that legacy. Anyway, AI "coding" makes me think of that but on steroids. It's fine, but the hype around it is silly, it's like declaring you can replace Microsoft Word because "New Project From Template" you got a little rich text widget in a window with a toolbar. One of the things mentioned in the article is the writer was confused that Claude's airplane was sideways. But it makes perfect sense, Claude doesn't really care about or understand airplanes, and as soon as you try to refine these New Project From Template things the AI quickly stops being useful.

aiauthoritydev 2y ago
Visual basic created a revolution in software world especially for poor countries like India. You will be surprised how many systems were automated and turned into software driven processes. It was just mindblowing. If AI driven software can do it on steroid it would be a massive impact on economy.
- overgard 2y ago
  I see what you're saying, but even with visual basic you needed to understand the basics of coding to get anywhere. Vibe coding seems to be more about intentionally not understanding things. I just think that, if you can code it yourself, maybe AI saves you some time, if you can't code it yourself, AI gives you a fancy toy that you can't expand or change. And God help you if you make an online service..

charcircuit 2y ago

>Minecraft-styled block buildings The buildings weren't minecraft style in either case. They weren't formed on a voxel grid and the textures weren't 16x16, but rather a rectangle or at least stretched to one. Also buildings typically are not just built as a cuboid.

sxp 2y ago

One prompt I use for testing is: "Using three.js, render a spinning donut with gl.TRIANGLE_STRIP". The catch here is that three.js doesn't support TRIANGLE_STRIP for architectural reasons[1]. Before I knew this, I got confused as to why all the AIs kept failing and gaslighting me about using TRIANGLE_STRIP. If the AI fails to tell the user that this is an impossible task, then it has failed the test. So far, I haven't found an AI that can determine that the request isn't valid. [1] https://discourse.threejs.org/t/is-there-really-no-way-to-use-gl-triangle-strip/53900 https://discourse.threejs.org/t/is-there-really-no-way-to-us...

nisten 2y ago

They nerfed it as of sunday March 30, a lot of people noticed performance drop and it rambling. https://x.com/nisten/status/1906141823631769983 https://x.com/nisten/status/1906141823631769983 Would be nice if this review actually wrote exactly when they conducted their test.

mtaras 2y ago
("it" being the Claude 3.7, not the Gemini)

breadwinner 2y ago

The loser in the AI model competition appears to be... Microsoft. When ChatGPT was the only game in town Microsoft was seen as a leader, thanks to their wise investment in Open AI. They relied on Open AI's model and didn't develop their own. As a result Microsoft has no interesting AI products. Copilot is a flop. Bing failed to take advantage of AI, Perplexity ate their lunch. Satya Nadella last year: “Google should have been the default winner in the world of big tech’s AI race”. Sundar Pichai's response: “I would love to do a side-by-side comparison of Microsoft’s own models and our models any day, any time. They are using someone else's model.” See: https://www.msn.com/en-in/money/news/sundar-pichai-vs-satya-nadella-google-ceo-dares-microsoft-boss-for-ai-face-off-any-day-any-time/ar-AA1vockF https://www.msn.com/en-in/money/news/sundar-pichai-vs-satya-...

dughnut 2y ago
Copilot is the only authorized AI at my company (50K FTE). I would be cautious to make any assumptions about how well anyone is doing in the AI space without some real numbers. My cynical opinion on enterprise software sales is that procurement decisions have absolutely nothing to do with product cost, performance, or value.
maxloh 2y ago
Note that Microsoft do have their own LLM team, and their own model called Phi-4. https://huggingface.co/microsoft/phi-4 https://huggingface.co/microsoft/phi-4
- VladVladikoff 2y ago
  Recently I was looking for a small LLM that could perform reasonably well while answering questions with low latency, for near realtime conversations running on a single RTX 3090. I settled on Microsoft’s Phi-4 model so far. However I’m not sure yet if my choice is good and open to more suggestions!
  - mywittyname 2y ago
    I've been using claude running via Ollama (incept5/llama3.1-claude) and I've been happy with the results. The only annoyance I have is that it won't search the internet for information because that capability is disabled via flag.

stared 2y ago

Just a moment ago I tried to use Gemini 2.5 (in Cursor) to use Python Gemini SDK. It failed, even with a few iterations. Then run Claude 3.7 - it worked fine. So yeah, depends on the case. But I am surprised that model creators don't put extra effort into dealing with setting their own tools.

Extropy_ 2y ago

Why is Grok not in their benchmarks? I don't see comparisons to Grok in any recent announcements about models. In fact, I see practically no discussion of Grok on HN or anywhere except Twitter in general.

nathanasmith 2y ago
Is there an API for Grok yet? If not that could be the issue.

raffkede 2y ago

I had huge success letting Gemini 2.5 oneshot whole codebases in a single text file format and then split it up with a script. It's putting in work for like 5 minutes and spits out a working codebase, I also asked it to show of a little bit and it almost one shotted a java cloud service to generate pdf invoices from API calls, (made some minor mistakes but after feeding them back it fixed them) I basically use two scripts one to flatten the whole codebase into one text file and one to split it, give it a shot it's amazing...

mvdtnz 2y ago
Anything that can fit in a single LLM output is not a "codebase" it's just a start. Far too many people with no experience in real software projects think their little 1800 line apps are representative of real software development.
archeantus 2y ago
Can you please expound on this? You’re using this approach to turn an existing codebase into a single file and then asking Gemini to make changes/enhancements? Does it also handle breaking the files back out? Would love more info!
- raffkede 2y ago
  I created a script that merges all files in a directory into this format, and a counterpart that splits it again. Below is just a small sample I asked it to create to show the format, but I did it with almost 80 files including lots of documentation etc. When providing the flat format it was able to replicate it without much instructions for a blank prompt i had success with the prompt below ===FILE=== Index: 1 Path: src/main/java/com/example/myapp/Greeter.java Length: 151 Content: package com.example.myapp; public class Greeter { public String getGreeting() { return "Hello from the Greeter class!"; } } ===ENDFILE=== ===FILE=== Index: 2 Path: src/main/java/com/example/myapp/Main.java Length: 222 Content: package com.example.myapp; public class Main { public static void main(String[] args) { Greeter greeter = new Greeter(); String message = greeter.getGreeting(); System.out.println("Main app says: " + message); } } ===ENDFILE=== ===FILE=== Index: 3 Path: pom.xml Length: 659 Content: <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>my-simple-app</artifactId> <version>1.0-SNAPSHOT</version> <properties> <maven.compiler.source>17</maven.compiler.source> <maven.compiler.target>17</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> </project> ===ENDFILE=== Prompt to request the format if starting from scratch: Present the entire codebase using the following multi-file format: The codebase should be presented as a single, monolithic text output. Inside this output, represent each file of the project individually using the following structure: Start Marker: Each file must begin with the exact line: ===FILE=== Metadata Block: Immediately following the start marker, include these four specific metadata lines, each on its own line: Index: <N> (where <N> is a sequential integer index for the file, starting from 1). Path: <path/to/file/filename.ext> (The full relative path of the file from the project's root directory, e.g., index.html, css/style.css, js/script.js, jobs.html, etc.). Length: <L> (where <L> is the exact character count of the file's content that follows). Content: (This literal line acts as a separator). File Content: Immediately after the Content: line, include the entire raw content of the file. Preserve all original line breaks, indentation, and formatting exactly as it should appear in the actual file. End Marker: Each file's section must end with the exact line: ===ENDFILE=== Ensure all necessary files for the project (HTML, CSS, JS) are included sequentially within the single output block according to this structure. Crucially, enclose the entire multi-file output, starting from the very first ===FILE=== line down to the very last ===ENDFILE=== line, within a single Markdown fenced code block using exactly five backticks (`````) on the lines immediately before the first ===FILE=== and immediately after the last `===ENDFILE===`. This ensures that any triple backticks (```) within the generated file content are displayed correctly.

phforms 2y ago

I like using LLMs more as coding assistents than have them write the actual code. When I am thinking through problems of code organization, API design, naming things, performance optimization, etc., I found that Claude 3.7 often gives me great suggestions, points me in the right direction and helps me to weigh up pros and cons of different approaches. Sometimes I have it write functions that are very boilerplate to save time, but I mostly like to use it as a tool to think through problems, among other tools like writing in a notebook or drawing diagrams. I enjoy programming too much that I’d want an AI to do it all for me (it also helps that I don’t do it as a job though).

anotherpaulg 2y ago

Gemini 2.5 Pro set a wide SOTA on the aider polyglot coding leaderboard [0]. It scored 73%, well ahead of the previous 65% SOTA from Sonnet 3.7. I use LLMs to improve aider, which is >30k lines of python. So not a toy codebase, not greenfield. I used Gemini 2.5 Pro for the majority of the work on the latest aider release [1]. This is the first release in a very long time which wasn't predominantly written using Sonnet. The biggest challenge with Gemini right now is the very tight rate limits. Most of my Sonnet usage lately is just when I am waiting for Gemini’s rate limits to cool down. [0] https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ [1] https://aider.chat/docs/faq.html#what-llms-do-you-use-to-build-aider https://aider.chat/docs/faq.html#what-llms-do-you-use-to-bui...

atonse 2y ago
As someone who just adopted Cursor (and MCP) 2-3 weeks ago, Aider seems like a different world. The examples of "create a new simple video game" cause me to glaze over. Do you have a screencast of how you use aider to develop aider? I'd love to see how a savvy expert uses these tools for real-world solutions.
- anotherpaulg 2y ago
  I actually get asked for screencasts a lot, so I made recently made some [0]. The recording of adding support for 100+ new coding languages with tree-sitter [1] shows some pretty advanced usage. It includes using aider to script downloading a collection of files, and using ad-hoc bash scripts to have aider modify a collection of files. [0] https://aider.chat/docs/recordings/ https://aider.chat/docs/recordings/ [1] https://aider.chat/docs/recordings/tree-sitter-language-pack.html https://aider.chat/docs/recordings/tree-sitter-language-pack...
  - atonse 2y ago
    This is perfect. Thank you!
  - sedgjh23 2y ago
    This is excellent, thank you.

InTheArena 2y ago

The amazing bit about claude code is it's ability to read code, and fit into the existing code base. I tried visual studio code w/ roo, and it blew up my 50 daily request limit immediately. Any suggestions on better tooling for a claude code like experience with Gemeni 2.5 pro?

sfjailbird 2y ago

Every test task, including the coding test, is a greenfield project. Everything I would consider using LLMs for is not. Like, I would always need it to do some change or fix on a (large) existing project. Hell, even the examples that were generated would likely need subsequent alterations (ten times more effort goes into maintaining a line of code than writing it). So these tests are meaningless to me, as a measure of how useful these models are. Great for comparison with each other, but would be interesting to include some tests with more realistic work.

maxnevermind 2y ago
Indeed, I surprised to see that is has been in top-10 on HN for today. I thought everyone already realized that all of those examples like "create a flappy bird game" are not realistic and do not reflect the actual usefulness of the model, very few professionals in the industry endlessly create flappy bird games for a living.