3 ms·

For coding, I like the Aider polyglot benchmark, since it covers multiple programming languages. Gemini 2.5 Pro got 72.9% o3 high gets 81.3%, o4-mini high get

For coding, I like the Aider polyglot benchmark, since it covers multiple programming languages. Gemini 2.5 Pro got 72.9% o3 high gets 81.3%, o4-mini high gets 68.9%

asadm 1y ago
thanks
vessenes 1y ago
where do you find those o3 high numbers? https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ currently has gemini 2.5 pro as the leader at, as you say, 72.9%.
- re-thc 1y ago
  It's in the OpenAI article post (OP) i.e. OpenAI ran Aider themselves.
  - vessenes 1y ago
    Update: the leaderboard has o3 high + 4o tops of the charts now with 82.7%. This is a) amazing b) 20x more expensive than Gemini.
croemer 1y ago
Isn't it easy to train on the specific Exercism exercises that this benchmark uses?
jumpCastle 1y ago
It was a good benchmark until it entered the training set.