3 ms·

I’m super surprised that C++ scores so high, this does not match our experience at all, and for anything performance critical it always drops the ball completel

I’m super surprised that C++ scores so high, this does not match our experience at all, and for anything performance critical it always drops the ball completely. I also don’t understand how these “games” map to real world complex problems. How are you measuring success? How does “adversarial customer service” map to “this LLM is better at C++ than the other” ? How are you sure you’re not just benchmarking language suitability for a problem ? I have so many questions about this…

gertlabs 5mo ago
- The majority of the environments can be played where the agent writes code to work the environment towards a goal. So the model is problem solving, and it has to do so in a particular language, and some languages outperform others. We have a lot of data to back up the improved compiled language performance, but note these are for successful code submissions (failures are counted in a different metric). With the Languages chart we're moreso measuring how good the ideas they came up with were, once they already compiled/didn't fail basic environment rules. - You need to run evals at scale to converge on this kind of behavior: these benchmarks run samples across a pool of hundreds of different types of environments - Some games are too open-ended to support code play. The customer service game is an example of that, where models are called on every tick of the environment to make a decision (that's the 'decision making' part of the evals which is weighted lowest). Very interesting results but not testing coding ability, just general reasoning. Not sure what issues you have with models writing C++ vs other languages, but I can imagine all sorts of C++ specific bottlenecks not directly related to the model's ability to reason in the language, like the dependencies, verbosity, extra effort to manage memory, etc. I have only done a little C/embedded work since agentic coding happened but I was pleasantly surprised.
- bdamm 5mo ago
  I've found the current cream of the crop to be quite good at resource management. I've sic'd Opus on some very gnarly lambda context bugs and it has directly improved the stability of the product I'm working on right now in a very substantial way. It couldn't quite do it entirely by itself, but with the right nudges here and there, it has absolutely accellerated the debugging work. It is particularly good at analyzing crashes and piecing together the detective work of what preconditions must exist for certain crashes to occur.
- stingraycharles 5mo ago
  I think my problem is that I’m not sure I understand whether you evals are testing language abilities or reasoning abilities. It seems to present results as if they’re testing language abilities, but the problems seem to be reasoning problems.
  - gertlabs 5mo ago
    I'm not so sure there's a difference. The main thing we want to measure for LLMs is broad reasoning capability, but seeing how that ability changes under different constraints (like programming language) is the interesting part.
    - stingraycharles 5mo ago
      Ok, when you put it that way, I agree that this is in fact interesting. Maybe you should actually put that sentence in some kind of form on your website because it clarifies a lot.
      - gertlabs 5mo ago
        Appreciate the feedback, we're taking a lot of good recommendations from this thread, which will be live in a few days: better explanations/tooltips, more samples for any filter sets we offer (or show sample counts if potentially noisy), and adding some functional languages like Clojure
shimman 5mo ago
It's a VC backed company using "scoreboards" as a means of marketing. There is nothing scientific or academic about this, stop assuming as much.
- stingraycharles 5mo ago
  You’re probably correct, but the problem I have is that even a lot of “scientific” benchmarks have similar issues. The whole AI benchmark landscape seems to suffer from similar issues, that benchmarks don’t appear to be matching real world experience (eg open weight Chinese models scoring super high on all benchmarks, while clearly being inferior when using in real life).
  - shimman 5mo ago
    No offense, but honestly look into who are making these "benchmarks" it's all industry insiders little to no academics to be found. If you listen to actual academics on the subject they are far more realistic than what VC fools will tell you otherwise, which makes sense: one has an incentive to further the knowledge of humanity while the other wants to rat fuck the world to make a quick buck.

gertlabs 5mo ago

- The majority of the environments can be played where the agent writes code to work the environment towards a goal. So the model is problem solving, and it has to do so in a particular language, and some languages outperform others. We have a lot of data to back up the improved compiled language performance, but note these are for successful code submissions (failures are counted in a different metric). With the Languages chart we're moreso measuring how good the ideas they came up with were, once they already compiled/didn't fail basic environment rules. - You need to run evals at scale to converge on this kind of behavior: these benchmarks run samples across a pool of hundreds of different types of environments - Some games are too open-ended to support code play. The customer service game is an example of that, where models are called on every tick of the environment to make a decision (that's the 'decision making' part of the evals which is weighted lowest). Very interesting results but not testing coding ability, just general reasoning. Not sure what issues you have with models writing C++ vs other languages, but I can imagine all sorts of C++ specific bottlenecks not directly related to the model's ability to reason in the language, like the dependencies, verbosity, extra effort to manage memory, etc. I have only done a little C/embedded work since agentic coding happened but I was pleasantly surprised.

bdamm 5mo ago
I've found the current cream of the crop to be quite good at resource management. I've sic'd Opus on some very gnarly lambda context bugs and it has directly improved the stability of the product I'm working on right now in a very substantial way. It couldn't quite do it entirely by itself, but with the right nudges here and there, it has absolutely accellerated the debugging work. It is particularly good at analyzing crashes and piecing together the detective work of what preconditions must exist for certain crashes to occur.
stingraycharles 5mo ago
I think my problem is that I’m not sure I understand whether you evals are testing language abilities or reasoning abilities. It seems to present results as if they’re testing language abilities, but the problems seem to be reasoning problems.
- gertlabs 5mo ago
  I'm not so sure there's a difference. The main thing we want to measure for LLMs is broad reasoning capability, but seeing how that ability changes under different constraints (like programming language) is the interesting part.
  - stingraycharles 5mo ago
    Ok, when you put it that way, I agree that this is in fact interesting. Maybe you should actually put that sentence in some kind of form on your website because it clarifies a lot.
    - gertlabs 5mo ago
      Appreciate the feedback, we're taking a lot of good recommendations from this thread, which will be live in a few days: better explanations/tooltips, more samples for any filter sets we offer (or show sample counts if potentially noisy), and adding some functional languages like Clojure

shimman 5mo ago

It's a VC backed company using "scoreboards" as a means of marketing. There is nothing scientific or academic about this, stop assuming as much.

stingraycharles 5mo ago
You’re probably correct, but the problem I have is that even a lot of “scientific” benchmarks have similar issues. The whole AI benchmark landscape seems to suffer from similar issues, that benchmarks don’t appear to be matching real world experience (eg open weight Chinese models scoring super high on all benchmarks, while clearly being inferior when using in real life).
- shimman 5mo ago
  No offense, but honestly look into who are making these "benchmarks" it's all industry insiders little to no academics to be found. If you listen to actual academics on the subject they are far more realistic than what VC fools will tell you otherwise, which makes sense: one has an incentive to further the knowledge of humanity while the other wants to rat fuck the world to make a quick buck.