gertlabs

9 ms·

121.
by gertlabs 6mo ago
Take a look at the Time column in https://gertlabs.com/?mode=oneshot_coding -- this is the total time to complete a solution for a reasonably complex problem end-to-end (you would have to divide by avg submission size to es
122.
by gertlabs 6mo ago
Waiting on public API release. Once it drops, results will be up within 24 hours.
123.
by gertlabs 6mo ago
Objective, detailed benchmark results at https://gertlabs.com Early takeaways: from this release, DeepSeek V4 Flash is the model to pay attention to here. It's cheap, effective, and REALLY fast. The Pro model is slow, not
124.
by gertlabs 6mo ago
Good idea.
125.
by gertlabs 6mo ago
Not sure what you mean. Time series chart of model performance over time to see if proprietary models get degraded? That's in the works, but we will need a couple months more data collection before launch.
126.
by gertlabs 6mo ago
We've been working on a way to address the obvious problems with existing benchmarks, by creating a single comprehensive benchmark that measures things that technical people care about, while also getting as close to an objective, &quo
127.
Gbench Intelligence Benchmark(gertlabs.com)
4 points by gertlabs 6mo ago|1 comments
128.
by gertlabs 6mo ago
Update: Kimi K2.5 one-shot results are live. It wasn't a noteworthy release compared to K2.6: https://gertlabs.com/?mode=oneshot_coding
129.
by gertlabs 6mo ago
Thanks -- that one is categorized under Trading/Financial, whereas betting is reserved for games like Pot Limit Omaha Hilo. That's a good idea for a feature request, including the tags for the spectatable demo games.
130.
by gertlabs 6mo ago
Good question. We missed that release entirely. Our automated model checker only went live 2 months ago so they were manually curated prior to that. I'm adding it now. It'll be live in ~12 hours.
131.
by gertlabs 6mo ago
We will as soon as API access is widely available. Once a model goes live, we typically have one-shot reasoning benchmarks up in ~8 hours and comprehensive agentic/combined benchmarks up after 24-48 hours. We're working on buildin
132.
by gertlabs 6mo ago
We recently added cost (last week), so data is sparse. Check back in a few weeks and it will be represented somewhere on the homepage, probably in the Efficiency Chart at the bottom. We also plan to show model performance deviation over tim
133.
by gertlabs 6mo ago
I did not know about this! We've put a lot of effort into probing providers and their offerings and auto-selecting the best options. I wonder how well their exacto option works. Going to test it out, thanks!
134.
by gertlabs 6mo ago
It's interesting; I can only speculate as to the underlying reason. When given enough time, models outperform in Rust/C++ in longer agentic tasks, and actually perform worst in Python. For tasks that aren't judged on code spe
135.
by gertlabs 6mo ago
Early benchmarks show tremendous improvement over Kimi K2 Thinking, which didn't perform well on our benchmarks (and we do use best available quantization). Kimi K2.6 is currently the top open weights model in one-shot coding reasoning
136.
by gertlabs 6mo ago
This is real issue in our benchmarks. Beware of OpenRouter providers that don't specify quantizations or use lower ones than you might be expecting. OpenRouter does provide configuration options for this, and it often limits your optio
137.
by gertlabs 6mo ago
We calculate percentiles based on successful submissions only, and then apply success rate as a separate measurement, which is incorporated into our relative rankings. So we do penalize evals where the player failed the game, but not in the
138.
by gertlabs 6mo ago
We only have some basic time filtering ( https://gertlabs.com/?days=30 ), but most of our samples are from the last 2 months. This is a visualization we plan to add when we've collected more historical data. But we did h
139.
by gertlabs 6mo ago
Early benchmark results on our private complex reasoning suite: https://gertlabs.com/?mode=agentic_coding Opus 4.7 is more strategic, more intelligent, and has a higher intelligence floor than 4.6 or 4.5. It's roughly
140.
by gertlabs 6mo ago
In our benchmarks we exclusively use a custom harness for measuring tool capability. It has common tools that any harness would have, like a thin wrapper around shell commands, basic file editors, etc. but an important part of agentic intel
141.
Gemma 4 and the Economics of Selling AI(gertlabs.com)
6 points by gertlabs 6mo ago|0 comments
142.
by gertlabs 6mo ago
We add samples every week, so I'm curious if the numbers will move. They did a similar re-release during the Gemini 3.1 Pro Preview rollout, and released a custom-tools version with its own slug, which performs MUCH better on custom ha
143.
by gertlabs 6mo ago
In one shot coding, surprisingly, yes, by a decent amount. And it isn't a sample size issue. In agentic, no: https://gertlabs.com/?agentic=agentic My early takeaway is that Gemma 26B-A4B is the best tuned out of the bu
144.
by gertlabs 6mo ago
Gemma 4 26B really is an outlier in its weight class. In our little known, difficult to game benchmarks, it scored about as well as GPT 5.2 and Gemini 3 Pro Preview on one-shot coding problems. It had me re-reviewing our entire benchmarking
145.
by gertlabs 6mo ago
I partially agree, but C++ is the second best agentic language! (of 6 tested). LLMs are pretty good at reading machine output. My pet theory is that it has more to do with the training data in lower level languages being of a more interesti
146.
by gertlabs 6mo ago
Since programming is increasingly offloaded to LLMs and English is the main way engineers interact with code, it's interesting to see how LLMs reason in different programming languages. In our benchmarking, we've found LLMs perfor
147.
by gertlabs 6mo ago
The strongest model we've benchmarked on our comprehensive, little known, and difficult to game benchmark, is still Claude Opus 4.5 for agentic workflows. That's not a typo. Interpret that how you will, but if Anthropic had to tak
148.
Benchmark Fatigue(gertlabs.com)
3 points by gertlabs 6mo ago|0 comments
149.
by gertlabs 6mo ago
Interesting idea. The metric I'd intuitively want to see is low variance between harnesses for a smarter model. But if a large sample of models statistically outperformed with a certain harness, that's indeed a valuable signal for
150.
by gertlabs 6mo ago
We're still adding samples, but some early takeaways from benchmarking on https://gertlabs.com : Contrary to the model card, its one-shot performance is more impressive than its agentic abilities. On both metrics, GLM 5.1 is

More ›