5 ms·
Surprisingly, LLMs are actually much worse at reasoning in Python than other common programming languages for agentic coding tasks. Data here: https://gertlabs
by gertlabs 5mo ago
Surprisingly, LLMs are actually much worse at reasoning in Python than other common programming languages for agentic coding tasks.
Data here: https://gertlabs.com/rankings?mode=agentic_coding https://gertlabs.com/rankings?mode=agentic_coding
- hooloovoo_zoo 5mo agoMm, the code is constrained to run inside a game 'tick'?
- BariumBlue 5mo agoHah, I was just thinking that Python likely has a vast ocean of training data, but it's likely of lower quality, being much of it is written by beginners and those who aren't primarily programmers.
- topham 5mo agoThere's a broken idea that AI know Python because they're written in Python. Not how any of it works.
- gertlabs 5mo agoWhile recent models are capable of generalizing to any language at this point, I do think there are weights from their pretraining corpus that still leak through into how they create their responses. We observed similar language performance patterns across models from different providers, btw.
- dasyatidprime 5mo agoNot what anyone was talking about. Training corpus ≠ inference engine.
- stefanfisk 5mo agoReminds me of the time I asked Claude to write some Wordpress code for me. The results were…rough.
- dariusj18 5mo agoThat was the hardest part of learning PHP, all the code examples online were just awful.
- andai 5mo agoWorked on a PHP project once. Every time I asked why something was done a certain way the answer was "dunno, we copy pasted this code snippet." Certain popular PHP codebases appear to use a similar methodology.
- Sohcahtoa82 5mo agoIt's why I consider PHP to be "RCE as a Service". So much copy/pasted code, some of it REALLY bad, and PHP has a lot of foot-guns that can lead to RCE.
- FireBeyond 5mo agoAll my vibe coded projects (personal) are Go backend services, with Typescript/React frontend. And my thoughts were based on similar things. Like why I wouldn't use PHP for that, either.
- librasteve 5mo agoI was (pleasantly) surprised by Claude Code doing Raku - also with a limited training set (~2000 Stack Overflow, a bunch of Rosetta, 2,500 modules). I put this down to the quality of the code for the core community who are all frankly uber-gremlins.
- polytely 5mo agoYeah Raku feels so expressive and lovely to me with the help of an AI assistant. I've only done toy programs and scripts with it but it is actually so nice.
- kraf 5mo agoThat's what I'm thinking too. There is a lot of noise and I know teams where the majority of the people writing Python just have no idea what they're doing. I'm working with Clojure which is used mostly by senior engineers and it still blows my mind how well Claude writes software in it even though it's a fringe language. It's even able to pick up in-house DSLs written with macros.
- chamomeal 5mo agoWas looking for somebody to mention that LLMs are weirdly good at clojure! Have you tried hooking your LLM up to a repl? It’s crazy stuff. Type systems are a great feedback loop, but a LLM with a REPL is something else. Much more dangerous though!!
- smoe 5mo agoHaving used Python on and off for 20 years, my experience with LLMs writing Python has been mixed. I don’t think that’s necessarily because of a low-quality dataset, but rather because Python’s applications are so broad and the language has gone through several paradigm shifts over time: sync vs. async, typed vs. untyped, scientific Python looking very different from web application code, some people really wishing it were an FP language, and others doing the clean-architecture OOP onion soup. It has gotten so fragmented. Recently, I had a more pleasant experience using LLMs with Go. It reminds me a bit of Python 2.x, when the community seemed, in my view, more focused on embracing a stupid simple language, with everyone trying to write roughly similar "Pythonic" code.
- stingraycharles 5mo ago> Having used Python on and off for 20 years, my experience with LLMs writing Python has been mixed. I don’t think that’s necessarily because of a low-quality dataset, but rather because Python’s applications are so broad and the language has gone through several paradigm shifts over time If there’s one language that is the prime example of this, it’s C++, and according to this benchmark it ranks incredibly high. I’m also thoroughly confused why Kimi 2.6 scores 83% while Opus 4.7 scores 67% for C++, GPT5.5 isn’t even in the top10. Gemma 4 31B scores 100% success rate for Python (!!) while Opus 4.6 only 65%. This benchmark really seems to be all over the place and doesn’t make sense.
- gertlabs 5mo agoThe more filters you apply (single model and single language, especially if you also filter by pipeline like agentic vs one-shot), the fewer samples, so there is variance. Known limitation that is inevitable with any finite budget. This is why we are selective about adding more languages because it will dilute the amount of samples we can run per language per model. But the aggregated statistics hold up well and are very consistent in our testing.
- stingraycharles 5mo agoI just applied a single filter, programming language.
- rossjudson 5mo agoMy standard joke here: Q: Say, what does this Python code do? A: Nobody f&%^ing knows.
- thfuran 5mo agoThat’s Perl.
- bushbaba 5mo agoCool to see my hunch be backed by data. Python is a scripting language with OOP bolted on. Means there’s not really a styling consistency that other languages have, with things tending to look like PHP, a collection of various scripts that invoke one another
- toxik 5mo agoPython was designed with objects in mind from day one.
- regularfry 5mo ago"Designed" is doing a lot of work here. There are clearly bits that are just bolted on because they didn't want to change the syntax.
- nsbk 5mo agoEVERYTHING in Python is an object. I’m not sure how that could have been bolted onto the language
- Daishiman 5mo agoThey don't tend to look like that. Grab any one of the packages people use in Python at is generally well-disciplined OOP code. There are teams of people who don't come from an engineering background who do utilize Python as a series of scripts with some extra sugar. Just because you can do that doesn't mean that you should.
- isityettime 5mo agoI would love to see how they do with functional languages and especially Lisps here. I've noticed pretty good performance with Emacs Lisp relative to overall model strength, but I haven't used LLMs to application code in any such languages. It would also be interesting to see how Python compares to other languages in its niche (Ruby, Perl, Raku). Thanks for putting this together! It's interesting.
- gertlabs 5mo agoThat's a good idea. Would you rather see Lisp or Scala? Any interest in Prolog? We are trying to be selective to keep the data concentrated, but we will eventually add a couple more, most likely to sample different programming paradigms.
- phillc73 5mo agoJust last night I was going down the rabbit hole of "what's the best programming language to use for vibe coding." I came to a short list of: a) Typed Racket b) OCaml c) Julia I would love to see those three added to your benchmarks. And Mistral Medium 3.5 added to the LLM list, please.
- gertlabs 5mo agoThanks for the recs, we will look into adding some of these, maybe OCaml for variety. I'm not familiar with Racket. Mistral Medium 3.5 is on there, but you will have to scroll down pretty far to find it (does not perform well): https://gertlabs.com/rankings?mode=oneshot_coding https://gertlabs.com/rankings?mode=oneshot_coding
- regularfry 5mo agoWhat's going on with Qwen3.6 27b? Filtered to Python it comes out at the top of the list, which seems... well, unlikely.
- 5mo ago
- ricardo_lien 5mo agoThis surprised me, but I can understand it - Python sucks in many ways lol.
- altmanaltman 5mo agoHey they said it had a lot of training data, not necessarily high-quality python code training data.
- w0m 5mo agoHuh. This surprises me. Digging, it seems it looks like it comes down to interpreted + dynamically typed vs compiled and statically typed. TIL. If i were to start a truly vibe project; Go would have a significant leg up.
- dnautics 5mo agoand yet dynamically typed elixir wipes the floor with go. https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/blob/main/figures/exp_acb.png https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/blob/ma...
- bontaq 5mo agoLLMs get ridiculous with elixir, especially with the repl, runtime, and ability to hot reload / directly test functions. It's really surprising to me it hasn't caught on more but I guess you have to see it to believe it.
- cultofmetatron 5mo agobuilt my startup in elixir and can concur. elixir has a relatively consistent syntax that makes for a pretty good target for llms. In my opinion, the only thing holding elixir back as an llm deliverable is that there's not as much training data for llms to work with. Of course if we had a new AI that could be trained on a minimum of existing training data, common lisp would absolutely beat out everything else. everything you mentioned about elixir (repl, runtime, and ability to hot reload / directly test functions) are possible and were invented in lisp with an AST instead of a syntactic language as the ultimate build artifact. CL lets you recover from exceptions and rewind the stack before reloading your fixes and continuing. I can't even fathom the workloads an LLM could conceive of working with that.
- nojito 5mo agoThis doesn't say what you think it does. Language like elixir were given easier problems to solve when compared to other languages.
- goodmattg 5mo ago[dead]
- js8 5mo agoThe LLMs are generally still pretty bad at (deductive) reasoning. IME they go along more with the things like variable names and comments than the actual program logic (it would be an interesting experiment to compare LLM's understanding of three identical programs with different identifiers, one with normal identifiers, one with obfuscated identifiers, and one with deliberately misleading identifiers). I also think this particular comparison comes down to typing, which helps to avoid LLM's reasoning go astray. When we reason we need to typically propagate the constraints to arrive at a solution to these constraints. I think the best language to reason in could be something like Lean, which allows both constraints and actual code to be expressed at the same time. Although this might not be the case for current LLMs, as I explain above.
- by364 5mo agowait till you look inside a neural network and realize they're incapable of deductive reasoning! amazing how many devs that talk about "AI" would probably have a hard time telling apart deductive and inductive reasoning.
- kelseyfrog 5mo agoHow so?
- js8 5mo agoThat's actually untrue. Yes, training a neural network is mostly inductive reasoning process. However, the ability of LLMs to reason deductively (as a chain of thought, although it's probably not the only mechanism) is an emergent phenomenon, rising up from the training it on data and problems that exhibit deductive reasoning. But of course, because the deductive reasoning is inductively taught, there might be various shortcuts which compromise the soundness of deductive reasoning. That's why my claim - LLMs are not as good at it as other algorithms, although they have many other strengths that make up for it.
- antonvs 5mo ago> wait till you look inside a neural network and realize they're incapable of deductive reasoning! You could just as well say, wait until you look inside a human brain and realize they’re incapable of deductive reasoning! Of course we’re not actually able to do that, but if you could you might very well find a similar apparent lack of whatever it is you expect to see at that level.
- fulafel 5mo agoWhat would comparing rates across languages tell in the context of this benchmark? Are the tasks the same or robustly difficulty-normalized across the languages? Also somehow the 2 language comparison graphs (avg percentile and success rate) rank Python in dramatically different positions, with Python outranking Rust and Java in the success rate. What does the avg percentile mean in this context?
- gertlabs 5mo agoSuccess rate measures the amount of code submissions that played the game/environment without failing (compilation, breaking game rules, violating sandbox, etc.), so it makes sense Python would do better there. Percentile compares only the submissions that didn't hard-fail. So they are a bit different, and we incorporate them both into the combined score.
- Yokohiii 5mo agoComparing rust to javascript, the gscore is rather similar in distribution, while python falls off. I don't see why python should be so much worse?
- robot-wrangler 5mo ago> Data here: https://gertlabs.com/rankings?mode=agentic_coding https://gertlabs.com/rankings?mode=agentic_coding Oh wow, we got "tribal domination", "market simulator" and "adversarial customer service". I don't know what those are but it sure sounds like big torment nexus milestones Maybe we could at least play nicer games like hackenbush and act surprised when there's some wicked use-case that's isomorphic. EDIT: Ok fine. I like "Rubik's Cube Chess" a lot. Never heard of it, is this analyzed formally at all? Hard to search for since there's tons of collisions
- gertlabs 5mo agoNot formally analyzed -- in practice, we see a lot of repetition/draws from code submissions. Our version is custom, and uses more pawns, which can move in any direction but don't upgrade to other pieces. We try to include just as many cooperatives games as competitive games, but both are important for measuring model ability in the real world.
- stingraycharles 5mo agoI’m super surprised that C++ scores so high, this does not match our experience at all, and for anything performance critical it always drops the ball completely. I also don’t understand how these “games” map to real world complex problems. How are you measuring success? How does “adversarial customer service” map to “this LLM is better at C++ than the other” ? How are you sure you’re not just benchmarking language suitability for a problem ? I have so many questions about this…
- gertlabs 5mo ago- The majority of the environments can be played where the agent writes code to work the environment towards a goal. So the model is problem solving, and it has to do so in a particular language, and some languages outperform others. We have a lot of data to back up the improved compiled language performance, but note these are for successful code submissions (failures are counted in a different metric). With the Languages chart we're moreso measuring how good the ideas they came up with were, once they already compiled/didn't fail basic environment rules. - You need to run evals at scale to converge on this kind of behavior: these benchmarks run samples across a pool of hundreds of different types of environments - Some games are too open-ended to support code play. The customer service game is an example of that, where models are called on every tick of the environment to make a decision (that's the 'decision making' part of the evals which is weighted lowest). Very interesting results but not testing coding ability, just general reasoning. Not sure what issues you have with models writing C++ vs other languages, but I can imagine all sorts of C++ specific bottlenecks not directly related to the model's ability to reason in the language, like the dependencies, verbosity, extra effort to manage memory, etc. I have only done a little C/embedded work since agentic coding happened but I was pleasantly surprised.
- bdamm 5mo agoI've found the current cream of the crop to be quite good at resource management. I've sic'd Opus on some very gnarly lambda context bugs and it has directly improved the stability of the product I'm working on right now in a very substantial way. It couldn't quite do it entirely by itself, but with the right nudges here and there, it has absolutely accellerated the debugging work. It is particularly good at analyzing crashes and piecing together the detective work of what preconditions must exist for certain crashes to occur.
- riedel 5mo agoMy feeling is that for agentic tasks this is not only language design but also LSPs, error messages and static analysis capabilities that dominate the benchmarks. It would IMHO be interesting to look into better subsets of python and style/rewrite techniques as well as alternative linter and their effects on performance.
- gtrealejandro 5mo ago[dead]
- kevinautumn 5mo agoA strict compiler is basically a free feedback loop for the LLM.
- andai 5mo agoAlso the human. (I like being told about my bugs when I write them, instead of at some generally much more unpleasant moment in the future.)
- andai 5mo agoBut then why does JS score 50% better? (Almost identical to TypeScript.) Actually, JS can get a surprising amount of "intellisense" as well. Not sure if that was used here though.
- andai 5mo agoI thought it might have to do with the type system, but JavaScript type system is atrocious and it scores about 50% higher. So my theory does not make much sense.
- doug_durham 5mo agoReasoning??? Since when to LLMs reason. The word "reasoning" has become almost worthless in modern computer science due to misuse.
- antonvs 5mo agoLLMs can regularly produce correct conclusions from the input they’re given. When a human does that, we call the process involved “reasoning”. LLMs can even explain how their conclusions are supported in ways that again, if a human employed them, would be called reasoning. As such, functionally speaking - focusing on inputs and outputs - models can be said to reason. If you’re tempted to argue that it’s “only statistics” or similar, consider that human reasoning is only chemical and electrical signals in neurons.
- rightbyte 5mo agoI think I've read some study that the LLM reasoning is a mirage.