3 ms·
While I'm generally sympathetic to the idea that the public coding benchmarks are inadequate, to the point that I've written my own in the past and will likely
by jbellis 12d ago
While I'm generally sympathetic to the idea that the public coding benchmarks are inadequate, to the point that I've written my own in the past and will likely do so again, the complaint here is that "these tasks don't match what I do in my day job" which is ~always going to be the case. The hope with benchmarks is that you can capture properties that generalize, from examining performance against small set of tasks, and I do think that this is at least directionally true for well-designed evals.
(There's https://withspecific.com/benchmarks/real-swe https://withspecific.com/benchmarks/real-swe but since the tasks are private we still don't really know what they're representative of.)
- Zigurd 12d agoI don't mean to make you write a dissertation but to say that AI benchmarks can "capture properties that generalize from examining performance against a small set of tasks" is a bald assertion. It's a hypothesis without a theory behind it. I can profile some code and then tell you where the slow parts are. Unless there's some explanatory power to an AI benchmark, it's a bit like benchmarking pillows visually.
- menaerus 12d agoI started using gemini with caution given the "much worse" benchmarking points it has gotten and still does but in practice there's very little evidence I found in comparison to claude models. It performs really well on non trivial tasks.