3 ms·
Gemini 2.5 Pro set the SOTA on the aider polyglot coding leaderboard [0] with a score of 73%. This is well ahead of thinking/reasoning models. A huge jump from
by anotherpaulg 2y ago
Gemini 2.5 Pro set the SOTA on the aider polyglot coding leaderboard [0] with a score of 73%.
This is well ahead of thinking/reasoning models. A huge jump from prior Gemini models. The first Gemini model to effectively use efficient diff-like editing formats.
[0] https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- aoeusnth1 2y agoAm I correct in assuming that accuracy < using correct edit format? i.e. it made mistakes in 27% of the problems, 11% of which were due to (at least) messing up the diff format? In which case, google should be working on achieving better output format following, as Claude and R1 are able to hit nearly 100% accuracy on the format.
- anotherpaulg 2y agoIt does have fairly low adherence to the edit format, compared to the other frontier models. But it is much better than any previous Gemini model in this regard. Aider automatically asks models to retry malformed edits, so it recovers. And goes on to produce a SOTA score.
- aoeusnth1 2y agoOk, thanks for clearing that up.
- sagarpatil 2y agoThe only benchmark I care about. Thanks!