2 ms·

Any updated Pareto frontier graphs? https://paraplouis.github.io/llm-pareto-frontier/ https://paraplouis.github.io/llm-pareto-frontier/ is quite out of date now

Any updated Pareto frontier graphs? https://paraplouis.github.io/llm-pareto-frontier/ https://paraplouis.github.io/llm-pareto-frontier/ is quite out of date now.

tao_oat 3mo ago
I generally rely on LMArena for this: https://arena.ai/leaderboard/code/webdev/pareto https://arena.ai/leaderboard/code/webdev/pareto But it does take some days after model release before they collect enough data.
- Dibes 3mo ago
  Odd that open AI models aren't on that graph but are on the rankings! Must be a data lag issue?
- mdasen 3mo ago
  LMArena's "code" leaderboard is really skewed since it's a front-end JS code and design leaderboard. It generates a demo app with two models and then asks "do you prefer A or B". People can look at the code, but most of the time it's just going to be which one looks nicer. Models that people like the design aesthetic of (Claude, GLM) tend to do better in LMArena than they do on other benchmarks. Design matters, but you look at a model like GPT-5.5 and it's behind Kimi K2.6, Sonnet 4.6, Qwen3.7 Max, and GLM-5.1 on LMArena's code leaderboard. Then you look at benchmarks like DeepSWE and GPT-5.5 blows them out of the water with only Fable and GPT-5.6 beating it. I'm not saying that the LMArena leaderboard isn't useful, but I'm not sure how much weight I'd give it as a "code" leaderboard. I think often times it's a design comparison of simple front-end React apps rather than a coding comparison. GLM-5.2 is a very good model, but when you look at DeepSWE or Terminal-Bench v2, GPT-5.5 is well ahead.
Bromeo 3mo ago
openrouter->rankings shows a pareto frontier. https://openrouter.ai/rankings#benchmarks https://openrouter.ai/rankings#benchmarks
1899-12-30 3mo ago
you can get a rough version via artificialanalysis's cost per task https://artificialanalysis.ai/?cost=intelligence-vs-cost-per-task https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...

Any updated Pareto frontier graphs? https://paraplouis.github.io/llm-pareto-frontier/ https://paraplouis.github.io/llm-pareto-frontier/ is quite out of date now.

tao_oat 3mo ago

I generally rely on LMArena for this: https://arena.ai/leaderboard/code/webdev/pareto https://arena.ai/leaderboard/code/webdev/pareto But it does take some days after model release before they collect enough data.

Dibes 3mo ago
Odd that open AI models aren't on that graph but are on the rankings! Must be a data lag issue?
mdasen 3mo ago
LMArena's "code" leaderboard is really skewed since it's a front-end JS code and design leaderboard. It generates a demo app with two models and then asks "do you prefer A or B". People can look at the code, but most of the time it's just going to be which one looks nicer. Models that people like the design aesthetic of (Claude, GLM) tend to do better in LMArena than they do on other benchmarks. Design matters, but you look at a model like GPT-5.5 and it's behind Kimi K2.6, Sonnet 4.6, Qwen3.7 Max, and GLM-5.1 on LMArena's code leaderboard. Then you look at benchmarks like DeepSWE and GPT-5.5 blows them out of the water with only Fable and GPT-5.6 beating it. I'm not saying that the LMArena leaderboard isn't useful, but I'm not sure how much weight I'd give it as a "code" leaderboard. I think often times it's a design comparison of simple front-end React apps rather than a coding comparison. GLM-5.2 is a very good model, but when you look at DeepSWE or Terminal-Bench v2, GPT-5.5 is well ahead.

Bromeo 3mo ago

openrouter->rankings shows a pareto frontier. https://openrouter.ai/rankings#benchmarks https://openrouter.ai/rankings#benchmarks

1899-12-30 3mo ago

you can get a rough version via artificialanalysis's cost per task https://artificialanalysis.ai/?cost=intelligence-vs-cost-per-task https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...