4 ms·
I've been having some very impressive results from Gemini 2.5 Pro for complex coding tasks in the few hours I've been experimenting with it so far. I added a s
by simonw 2y ago
I've been having some very impressive results from Gemini 2.5 Pro for complex coding tasks in the few hours I've been experimenting with it so far.
I added a section about that to my review last night describing two of the larger examples: https://simonwillison.net/2025/Mar/25/gemini/#update-it-s-very-good-at-code https://simonwillison.net/2025/Mar/25/gemini/#update-it-s-ve...
(It's always risky saying anything like this on a forum like Hacker News because it's inevitable someone will find a way to argue that the examples are trivial/unrealistic/show I don't know what I'm doing/clearly just regurgitated from StackOverflow/etc, but I'll take the risk anyway.)
- intellectronica 2y agoSo far everything I tried indicates that it is superb at coding. Maybe the best yet (though I understand that some of the published benchmarks dispute that). Here it one-shotted a fully functional LISP interpreter for me: https://everything.intellectronica.net/p/the-little-lisper https://everything.intellectronica.net/p/the-little-lisper
- arresin 2y agohmm but how many examples of that exist on the internet practically verbatim?
- intellectronica 2y agoNot sure that's relevant. Obviously an LLM has to learn from something, but it's not a database. I could also program this myself and I don't think that it's an argument against my coding abilities that I have read the source code to many existing interpreters. I can only do it because I not only read but also understood and internalised.
- jgalt212 2y agoBut it's almost a database. See the full glass of wine conundrum.
- sebzim4500 2y agoThat seems to only have affected standalone diffusion models. E.g. gpt-4o native image generation is easily able to generate a full glass of wine.
- jgalt212 2y agonot a satisfactory version, from what I saw posted here yesterday. https://news.ycombinator.com/item?id=43475314 https://news.ycombinator.com/item?id=43475314
- wat10000 2y agoIt’s not a database, it’s a prediction engine. It’s going to be very good at predicting things it was trained on.
- hadlock 2y agoGood point. Turns out OpenAI's llms are pretty decent at coding x86_64 bios bootloaders in assembly, but as soon as you go off script from the two main examples online, it falls apart really quickly, as it's crystal clear it has no idea what is actually going on or the limitations of how bootloaders (and 2, and 3 stage bootloaders) work.
- hatefulmoron 2y agoA while back I wanted to learn more about how direct threaded Forth implementations work. Just explanations, not code. I went back and forth with Claude for a while, and I noticed that its ability to give me a coherent explanation was terrible. However, it was able to give me perfectly fine x86 assembler to implement portions of it -- I knew it was perfectly fine because it was reproducing code from jonesforth, which I had open in another tab.
- mentalgear 2y agoThere's only a risk if these statements are meant to generalise the overall performance of a system, instead of what they are: arbitrary samples.
- simonw 2y agoIt's getting a little less risky these days, but in the past just saying "LLMs are good at writing code" was enough to spark a hundred comment flame war.
- philipwhiuk 2y agoSure, because you're making a broad, provably false statement rather than a statement on a highly specific task.
- jgalt212 2y agoSimon is so important in this space, and rightfully so, I wonder if these models overweight his blog in the training sets. Similar to how car manufacturers design to what they know will impress the car reviewers.
- jjani 2y ago100% guaranteed at least someone at Google tried the pelican SVG prompt before the public release. Doesn't mean they necessarily adjusted anything based on its results, but no chance they're not at least taking them into account, the budgets are far too high for them not to do so.
- mattlondon 2y agoAgree - seems like the rumours and mutterings were true: This model is very, very good. Quite a few happy people at Google today I bet. Which leads me to wonder, it's not like the Gemini 2 models were terrible either - they consistently were in top 5 if not top 3, now they've smashed past everything with a +40 elo. Are we starting to see Google apply their compute/resources/data/money to assert dominance? What next from the recently-pretty-quiet Open AI? Are we getting to the stage where well-funded startups like Anthropic et al simply cannot compete with "google-scale" for general purpose models and end up as coding-only niche models? Sure you can throw GPUs at the problem and burn more investor cash, but are Google starting to run away with it with their data and infrastructure advantages? Who even comes close when you factor in data? Meta are the only people I can think of, but their data must be quite narrow (basically social graph and short-form videos and ad click data?) Exciting times.
- intellectronica 2y agoThat sounds plausible. Google have two advantages: 1. They do their own capital allocation, 2. TPUs. That likely means that they can execute more training runs in parallel, experiment more, and release when they hit a crack of gold. Independent labs that depend on outside investment have to carefully trade off experiments. Hence Stargate.
- aoeusnth1 2y agoI think if they could have been on top earlier, they would have been. They’ve been struggling to catch anthropic and OpenAI’s lead and they finally did it (for now), probably due to TPU superiority plus some secret sauce of some kind. Good! More competition means better service for the consumer.
- _l7dh 2y agoHi Simon, just read your blog post, thanks for the wrap-up. Just curious, what did you use to make Gemini look at all of your codebase, Aider, something else?
- Jimpulse 2y agoHe mentions what he used here, https://simonwillison.net/2025/Mar/26/notes/ https://simonwillison.net/2025/Mar/26/notes/. https://github.com/simonw/files-to-prompt https://github.com/simonw/files-to-prompt https://llm.datasette.io/ https://llm.datasette.io/
- _l7dh 2y agoMany thanks! For the record, there is another useful tool in the same vein, that packs a repo to be given to an LLM https://github.com/yamadashy/repomix https://github.com/yamadashy/repomix.
- Jensson 2y ago> (It's always risky saying anything like this on a forum like Hacker News because it's inevitable someone will find a way to argue that the examples are trivial/unrealistic/show I don't know what I'm doing/clearly just regurgitated from StackOverflow/etc, but I'll take the risk anyway.) What is the risk? Do you lose anything? Its like saying its risky to say these AI are not useful since people might say you are just using it wrong etc.
- simonw 2y agoThe risk is that I end up wasting a bunch of my time arguing with people about it, because I can't let someone be wrong on the internet. https://xkcd.com/386/ https://xkcd.com/386/
- naveen99 2y agoLove the quote from reed: “My hack to-do list is empty because I built everything. I keep thinking of new things and knocking them out while watching a movie or something.” Reminds me of the time when I discovered GTD. Don’t worry, we will find a way to become overwhelmed again.