gertlabs

8 ms·

91.
by gertlabs 5mo ago
If you are referring to the parent post, yes, hard to draw conclusions from such a small sample size. For our testing, we use hundreds of different environments across disciplines, and it seems to line up with subjective experience better t
92.
by gertlabs 5mo ago
I think benchmarks are improving and will always have value, but it's the equivalent to someone's college and GPA for an entry level job application. It's a strong signal for a job, but the soft skills are sometimes going to
93.
by gertlabs 5mo ago
Test result summaries are openly available, test environments are not.
94.
by gertlabs 5mo ago
I'm not sure how many of our requests got routed to Fireworks -- for our testing, we set preferences for routing to providers with the highest advertised quantizations / highest reasoning mode support / or preferably the mode
95.
by gertlabs 5mo ago
This is awesome data! I've been wanting to measure how closely hype aligns to our results at https://gertlabs.com/rankings Subjectively, it seemed like DeepSeek V4 Pro had the highest hype/performance ratio (meani
96.
by gertlabs 5mo ago
I'm glad we're seeing a shift towards objectively scored tests. We've been doing this at scale at https://gertlabs.com/rankings , and although the author looks to be running unique one-off samples, it's n
97.
by gertlabs 6mo ago
DeepSeek V4 Flash is the most cost effective model we've tested. We had to really understand why it outperformed DeepSeek V4 Pro (although even on unreliable model cards, Flash was very close to Pro). Pro is slower and smarter in one-s
98.
by gertlabs 6mo ago
Pro is smarter in one-shot problems, but it struggles with custom tooling, and spends too much time trying to figure out our harness. We ran a lot of samples, so I can't make excuses for the model. Flash is truly the better option over
99.
by gertlabs 6mo ago
All models are moving towards more frequent and more efficient tool use, which should close the gap on post-knowledge cutoff problems. The only tradeoff I see is speed, and Grok 4.3 is currently taking the fast side of that tradeoff.
100.
by gertlabs 6mo ago
Grok 4.3 is a unique model in our tests. It's one of the fastest models, and its responses are far smaller/token dense than other models with comparable performance. However, its overall coding reasoning ability is not competitive
101.
by gertlabs 6mo ago
We're still running Grok 4.3 evals, since API access is now widely available. So far it looks like it's not the frontier model, but definitely worthy of mention. The field moves fast... The benchmarks and blog post will be updated
102.
State of the AI Frontier, April 2026(gertlabs.com)
6 points by gertlabs 6mo ago|1 comments
103.
by gertlabs 6mo ago
In our testing, the Pro version underperformed because it struggled in agentic tasks where we use a harness with custom tools it was not trained on. Mostly formatting issues. All of the other major releases in April, including the Flash ver
104.
by gertlabs 6mo ago
This is fascinating, and aligned with my experience working on harnesses. I bet there is still significant upside left on the table with that same model. There was a narrative last year by Anthropic that each new model release had them maki
105.
by gertlabs 6mo ago
This is the most underrated release we tested at https://gertlabs.com I'm surprised they open sourced it. It's very comparable with Kimi K2.6 performance-wise, and slightly better with tools. And it's cheaper.
106.
by gertlabs 6mo ago
C# testing is a new feature added a few days ago from HN comment suggestions, samples will continue growing. Most C# data is currently for non-agentic workloads: https://gertlabs.com/?mode=oneshot_coding
107.
by gertlabs 6mo ago
Decision making refers to the environments where the LLM is called on every tick (like games with social communication), examples here: https://gertlabs.com/spectate . Because GPT 5.5 just launched and those games take longe
108.
by gertlabs 6mo ago
It's a surprising result, and a lot of it stems from the Pro variant struggling with our custom harness in agentic tasks (whereas Flash does fine), as well as provider instability. Failed requests are not counted against the model in i
109.
by gertlabs 6mo ago
https://gertlabs.com already does this at scale. An industry-standard benchmark shouldn't be hosted or designed by a lab producing the models, regardless.
110.
by gertlabs 6mo ago
Much appreciated! MiMo V2.5 Pro is by far the most underrated recent release (probably because it wasn't open weights from the start).
111.
by gertlabs 6mo ago
A better benchmark needs to be objectively scored, have multi-disciplinary, breadth, and be scalable (no single correct answer). That's what we designed at https://gertlabs.com . We put a lot of thought into it, and kept it
112.
by gertlabs 6mo ago
I have not run one personally, but I love the idea. Reminds me of yoyo-evolve. My friend made this repo: https://github.com/dwolner/cosmic-insight
113.
by gertlabs 6mo ago
No the full prompt is not available in the UI, sorry.
114.
by gertlabs 6mo ago
Our philosophy is that you can design problems so that they can scale through a few release cycles by making environments more complex, with no known ceiling. The key for scalability is not having a single correct answer (even though Victor
115.
by gertlabs 6mo ago
All of our environments are tick based (with ticks of varying speeds), and this is explained in the prompt given to the models, along with the latest observation and a history of recent events/conversations/actions. So that does m
116.
by gertlabs 6mo ago
Self organizing systems is an area of research to which I think LLMs will contribute immensely. But as of now, even newer AI models are not particularly insightful. I'm always surprised by how suboptimal near-frontier LLMs are at colla
117.
by gertlabs 6mo ago
I see the value in that, but there are a few reasons that isn't on the immediate roadmap -- mainly, it shifts focus from measuring the model to measuring the harness. The agentic benchmark section you see on the site is comparable to h
118.
by gertlabs 6mo ago
Results are up. GPT 5.5 is a beast.
119.
by gertlabs 6mo ago
Comprehensive coding reasoning benchmark results for GPT 5.5 with max reasoning are up at https://gertlabs.com/ Live decision and heavier agentic evals will continue being uploaded for 24 hours but I don't expect its l
120.
by gertlabs 6mo ago
No, some benchmarks are definitely objective, but most can be easily gamed. For example, most of the benchmarks on the model cards: they have measurable answers that don't rely on a human judge (a human made the question, but the answe

More ›