gertlabs

7 ms·

61.
by gertlabs 5mo ago
Success rate is essentially loading/compilation success + ability to adhere to the environments' rules. For one-shot responses, the majority of failures are environmental/syntax, which naturally favors interpreted languages.
62.
by gertlabs 5mo ago
When you're working on something difficult that requires a model to reason intelligently, lower level and strongly typed languages often outperform on the same problems [0]. We have a few hypotheses about why, with a moderately high co
63.
by gertlabs 5mo ago
I liked Rust before running a benchmark, but the gap between how effectively most LLMs write in Rust vs Go was still surprisingly large to me (especially in agentic harnesses where they can fix the initial environment issues). I've bec
64.
by gertlabs 5mo ago
C# is one of the best languages we evaluated for LLM problem solving. In fact, many strongly typed, lower level languages perform well in our testing (this isn't to say that the solutions are more maintainable, but the LLMs come up wit
65.
by gertlabs 5mo ago
Even with the V4 Pro discount, the V4 Flash model gives you the best performance per unit dollar, and better performance overall for agentic, tool-heavy workloads. V4 Pro is smarter in one-shot reasoning, but at a significant speed differen
66.
by gertlabs 5mo ago
This is what we do at gertlabs.com - the foundation labs are actually starving for better data. Having quality data is not the same as having a lot of data. Human curated data / RLHF cannot scale to a 5T model and synthetic data pipeli
67.
by gertlabs 5mo ago
We've been really impressed with the performance of ~30B parameter class models and how close they are to the frontier from ~6-12 months ago, which begs the question, are the frontier labs really serving 10T parameter models? Seems unl
68.
by gertlabs 5mo ago
Taking into account that this is a flash model, it's a strong release. It's very fast and frontier-ish for the price. Raw intelligence is high for a flash model. But Google's problem has always been productization and tool us
69.
by gertlabs 5mo ago
One thing we found interesting is that Clojure's benchmark-wide performance was nuked by Anthropic models, whereas OpenAI and many other frontier lab releases reasoned quite well in Clojure. The individual language per-model data is no
70.
LLM Performance by Programming Language(gertlabs.com)
5 points by gertlabs 5mo ago|1 comments
71.
by gertlabs 5mo ago
Appreciate the feedback, we're taking a lot of good recommendations from this thread, which will be live in a few days: better explanations/tooltips, more samples for any filter sets we offer (or show sample counts if potentially
72.
by gertlabs 5mo ago
The more filters you apply (single model and single language, especially if you also filter by pipeline like agentic vs one-shot), the fewer samples, so there is variance. Known limitation that is inevitable with any finite budget. This is
73.
by gertlabs 5mo ago
Not formally analyzed -- in practice, we see a lot of repetition/draws from code submissions. Our version is custom, and uses more pawns, which can move in any direction but don't upgrade to other pieces. We try to include just as
74.
by gertlabs 5mo ago
I'm not so sure there's a difference. The main thing we want to measure for LLMs is broad reasoning capability, but seeing how that ability changes under different constraints (like programming language) is the interesting part.
75.
by gertlabs 5mo ago
This was an unexpected result, and it held up under large sample sizes.
76.
by gertlabs 5mo ago
The more filters applied (one-shot coding only, Python only), the more variation you can expect from fewer samples -- that being said, it really is a great model so it's probably not too far above where it would end up with infinite sa
77.
by gertlabs 5mo ago
Success rate measures the amount of code submissions that played the game/environment without failing (compilation, breaking game rules, violating sandbox, etc.), so it makes sense Python would do better there. Percentile compares only
78.
by gertlabs 5mo ago
The deeper you go into the filters (single models, cross correlated by specific languages), the smaller your sample sizes. A known limitation, tbh I doubt Mistral is better than GPT 5.5 at programming in any specific language and probably h
79.
by gertlabs 5mo ago
- The majority of the environments can be played where the agent writes code to work the environment towards a goal. So the model is problem solving, and it has to do so in a particular language, and some languages outperform others. We hav
80.
by gertlabs 5mo ago
Thanks for the recs, we will look into adding some of these, maybe OCaml for variety. I'm not familiar with Racket. Mistral Medium 3.5 is on there, but you will have to scroll down pretty far to find it (does not perform well): https:
81.
by gertlabs 5mo ago
That's a good idea. Would you rather see Lisp or Scala? Any interest in Prolog? We are trying to be selective to keep the data concentrated, but we will eventually add a couple more, most likely to sample different programming paradigm
82.
by gertlabs 5mo ago
While recent models are capable of generalizing to any language at this point, I do think there are weights from their pretraining corpus that still leak through into how they create their responses. We observed similar language performance
83.
by gertlabs 5mo ago
Surprisingly, LLMs are actually much worse at reasoning in Python than other common programming languages for agentic coding tasks. Data here: https://gertlabs.com/rankings?mode=agentic_coding
84.
by gertlabs 5mo ago
The small Qwen 3.6 models handle context a little better than Gemma 4, but Gemma 4 26B in particular has such small and efficient solutions which are really smart for its weight class. I was so impressed with its performance in our benchmar
85.
by gertlabs 5mo ago
We observed slightly smaller outputs over long horizon agentic coding for GPT 5.5, at a significant improvement in overall response scores. For one-shot coding responses, GPT 5.5 was actually more verbose than GPT 5.4, but again, the respon
86.
by gertlabs 5mo ago
If the goal is cost savings, a good compromise is DeepSeek V4 Flash. It'll handle context and agentic work better than any model that can fit on a laptop.
87.
by gertlabs 5mo ago
This is true, we have the numbers to back it up on https://gertlabs.com/rankings?mode=oneshot_coding (check out the efficiency chart too) GPT 5.5/5.4 are the smartest models, but at great token / code bloat cost.
88.
by gertlabs 5mo ago
I am curious about the model, but for the most part, we have access to the same models that you do and only test models with standalone API releases.
89.
by gertlabs 5mo ago
Games is loosely defined here, as we run the bench across hundreds of unique environments. For some, the models write code to play a game, either one-shot or via a harness where they can iterate and use tools. Some they play directly, makin
90.
by gertlabs 5mo ago
GLM-5V-Turbo is a model I wanted to like due to its speed and API reliability, but it didn't perform well in our coding and reasoning testing. More recent open source models have made it obsolete. GLM 5.1 is so many light years ahead o

More ›