4 ms·

Numbers for SWE-bench Verified, Aider Polyglot, cost per million output tokens, output tokens per second, and knowledge cutoff month/year: SWE Ai

Numbers for SWE-bench Verified, Aider Polyglot, cost per million output tokens, output tokens per second, and knowledge cutoff month/year: SWE Aider Cost Fast Fresh Claude 3.7 70% 65% $15 77 8/24 Gemini 2.5 64% 69% $10 200 1/25 GPT-4.1 55% 53% $8 169 6/24 DeepSeek R1 49% 57% $2.2 22 7/24 Grok 3 Beta ? 53% $15 ? 11/24 I'm not sure this is really an apples-to-apples comparison as it may involve different test scaffolding and levels of "thinking". Tokens per second numbers are from here: https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25/providers https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25/pr... and I'm assuming 4.1 is the speed of 4o given the "latency" graph in the article putting them at the same latency. Is it available in Cursor yet?

meetpateltech 2y ago
Yes, it is available in Cursor[1] and Windsurf[2] as well. [1] https://twitter.com/cursor_ai/status/1911835651810738406 https://twitter.com/cursor_ai/status/1911835651810738406 [2] https://twitter.com/windsurf_ai/status/1911833698825286142 https://twitter.com/windsurf_ai/status/1911833698825286142
- cellwebb 2y ago
  And free on windsurf for a week! Vibe time.
deleted 2y ago
[deleted]
tomjen3 2y ago
Its available for free in Windsurf so you can try it out there. Edit: Now also in Cursor
deleted 2y ago
[deleted]
jsnell 2y ago
https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ shows 73% rather than 69% for Gemini 2.5 Pro? Looks like they also added the cost of the benchmark run to the leaderboard, which is quite cool. Cost per output token is no longer representative of the actual cost when the number of tokens can vary by an order of magnitude for the same problem just based on how many thinking tokens the model is told to use.
- modeless 2y ago
  There are different scores reported by Google for "diff" and "whole" modes, and the others were "diff" so I chose the "diff" score. Hard to make a real apples-to-apples comparison.
  - jsnell 2y ago
    The 73% on the current leaderboard is using "diff", not "whole". (Well, diff-fenced, but the difference is just the location of the filename.)
    - modeless 2y ago
      Huh, seems like Aider made a special mode specifically for Gemini[1] some time after Google's announcement blog post with official performance numbers. Still not sure it makes sense to quote that new score next to the others. In any case Gemini's 69% is the top score even without a special mode. [1] https://aider.chat/docs/more/edit-formats.html#diff-fenced:~:text=primarily%20used%20with%20the%20Gemini%20family%20of%20models https://aider.chat/docs/more/edit-formats.html#diff-fenced:~...
      - jsnell 2y ago
        The mode wasn't added after the announcement, Aider has had it for almost a year: https://aider.chat/HISTORY.html#aider-v0320 https://aider.chat/HISTORY.html#aider-v0320 This benchmark has an authoritative source of results (the leaderboard), so it seems obvious that it's the number that should be used.
soheil 2y ago
Yes on both Cursor and Windsurf. https://twitter.com/cursor_ai/status/1911835651810738406 https://twitter.com/cursor_ai/status/1911835651810738406
anotherpaulg 2y ago
I just finished updating the aider polyglot leaderboard [0] with GPT-4.1, mini and nano. My results basically agree with OpenAI's published numbers. Results, with other models for comparison: Model Score Cost Gemini 2.5 Pro Preview 03-25 72.9% $ 6.32 claude-3-7-sonnet-20250219 64.9% $36.83 o3-mini (high) 60.4% $18.16 Grok 3 Beta 53.3% $11.03 * gpt-4.1 52.4% $ 9.86 Grok 3 Mini Beta (high) 49.3% $ 0.73 * gpt-4.1-mini 32.4% $ 1.99 gpt-4o-2024-11-20 18.2% $ 6.74 * gpt-4.1-nano 8.9% $ 0.43 Aider v0.82.0 is also out with support for these new models [1]. Aider wrote 92% of the code in this release, a tie with v0.78.0 from 3 weeks ago. [0] https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ [1] https://aider.chat/HISTORY.html https://aider.chat/HISTORY.html
- purplerabbit 2y ago
  What model are you personally using in your aider coding? :)
  - anotherpaulg 2y ago
    Mostly Gemini 2.5 Pro lately. I get asked this often enough that I have a FAQ entry with automatically updating statistics [0]. Model Tokens Pct Gemini 2.5 Pro 4,027,983 88.1% Sonnet 3.7 518,708 11.3% gpt-4.1-mini 11,775 0.3% gpt-4.1 10,687 0.2% [0] https://aider.chat/docs/faq.html#what-llms-do-you-use-to-build-aider https://aider.chat/docs/faq.html#what-llms-do-you-use-to-bui...
- pzo 2y ago
  Did you benchmarked combo: DeepSeek R1 + DeepSeek V3 (0324)? There is combo on 3rd place : DeepSeek R1 + claude-3-5-sonnet-20241022 and also V3 new beating claude 3.5 so in theory R1 + V3 should be even on 2nd place. Just curious if that would be the case
ilrwbwrkhv 2y ago
Yup GPT 4.1 isn't good at all compared to the others. I tried a bunch of different scenarios, for me the winners: Deepseek for general chat and research Claude 3.7 for coding Gemini 2.5 Pro experimental for deep research In terms of price Deepseek is still absolutely fire! OpenAI is in trouble honestly.
- torginus 2y ago
  One task I do is I feed the models the text of entire books, and ask them various questions about it ('what happened in Chapter 4', 'what did character X do in the book' etc.). GPT 4.1 is the first model that has provided a human-quality answer to these questions. It seems to be the first model that can follow plotlines, and character motivations accurately. I'd say since text processing is a very important use case for LLMs, that's quite noteworthy.

meetpateltech 2y ago

Yes, it is available in Cursor[1] and Windsurf[2] as well. [1] https://twitter.com/cursor_ai/status/1911835651810738406 https://twitter.com/cursor_ai/status/1911835651810738406 [2] https://twitter.com/windsurf_ai/status/1911833698825286142 https://twitter.com/windsurf_ai/status/1911833698825286142

cellwebb 2y ago
And free on windsurf for a week! Vibe time.

deleted 2y ago

[deleted]

tomjen3 2y ago

Its available for free in Windsurf so you can try it out there. Edit: Now also in Cursor

deleted 2y ago

[deleted]

jsnell 2y ago

https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ shows 73% rather than 69% for Gemini 2.5 Pro? Looks like they also added the cost of the benchmark run to the leaderboard, which is quite cool. Cost per output token is no longer representative of the actual cost when the number of tokens can vary by an order of magnitude for the same problem just based on how many thinking tokens the model is told to use.

modeless 2y ago
There are different scores reported by Google for "diff" and "whole" modes, and the others were "diff" so I chose the "diff" score. Hard to make a real apples-to-apples comparison.
- jsnell 2y ago
  The 73% on the current leaderboard is using "diff", not "whole". (Well, diff-fenced, but the difference is just the location of the filename.)
  - modeless 2y ago
    Huh, seems like Aider made a special mode specifically for Gemini[1] some time after Google's announcement blog post with official performance numbers. Still not sure it makes sense to quote that new score next to the others. In any case Gemini's 69% is the top score even without a special mode. [1] https://aider.chat/docs/more/edit-formats.html#diff-fenced:~:text=primarily%20used%20with%20the%20Gemini%20family%20of%20models https://aider.chat/docs/more/edit-formats.html#diff-fenced:~...
    - jsnell 2y ago
      The mode wasn't added after the announcement, Aider has had it for almost a year: https://aider.chat/HISTORY.html#aider-v0320 https://aider.chat/HISTORY.html#aider-v0320 This benchmark has an authoritative source of results (the leaderboard), so it seems obvious that it's the number that should be used.

soheil 2y ago

Yes on both Cursor and Windsurf. https://twitter.com/cursor_ai/status/1911835651810738406 https://twitter.com/cursor_ai/status/1911835651810738406

anotherpaulg 2y ago

I just finished updating the aider polyglot leaderboard [0] with GPT-4.1, mini and nano. My results basically agree with OpenAI's published numbers. Results, with other models for comparison: Model Score Cost Gemini 2.5 Pro Preview 03-25 72.9% $ 6.32 claude-3-7-sonnet-20250219 64.9% $36.83 o3-mini (high) 60.4% $18.16 Grok 3 Beta 53.3% $11.03 * gpt-4.1 52.4% $ 9.86 Grok 3 Mini Beta (high) 49.3% $ 0.73 * gpt-4.1-mini 32.4% $ 1.99 gpt-4o-2024-11-20 18.2% $ 6.74 * gpt-4.1-nano 8.9% $ 0.43 Aider v0.82.0 is also out with support for these new models [1]. Aider wrote 92% of the code in this release, a tie with v0.78.0 from 3 weeks ago. [0] https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ [1] https://aider.chat/HISTORY.html https://aider.chat/HISTORY.html

purplerabbit 2y ago
What model are you personally using in your aider coding? :)
- anotherpaulg 2y ago
  Mostly Gemini 2.5 Pro lately. I get asked this often enough that I have a FAQ entry with automatically updating statistics [0]. Model Tokens Pct Gemini 2.5 Pro 4,027,983 88.1% Sonnet 3.7 518,708 11.3% gpt-4.1-mini 11,775 0.3% gpt-4.1 10,687 0.2% [0] https://aider.chat/docs/faq.html#what-llms-do-you-use-to-build-aider https://aider.chat/docs/faq.html#what-llms-do-you-use-to-bui...
pzo 2y ago
Did you benchmarked combo: DeepSeek R1 + DeepSeek V3 (0324)? There is combo on 3rd place : DeepSeek R1 + claude-3-5-sonnet-20241022 and also V3 new beating claude 3.5 so in theory R1 + V3 should be even on 2nd place. Just curious if that would be the case

ilrwbwrkhv 2y ago

Yup GPT 4.1 isn't good at all compared to the others. I tried a bunch of different scenarios, for me the winners: Deepseek for general chat and research Claude 3.7 for coding Gemini 2.5 Pro experimental for deep research In terms of price Deepseek is still absolutely fire! OpenAI is in trouble honestly.

torginus 2y ago
One task I do is I feed the models the text of entire books, and ask them various questions about it ('what happened in Chapter 4', 'what did character X do in the book' etc.). GPT 4.1 is the first model that has provided a human-quality answer to these questions. It seems to be the first model that can follow plotlines, and character motivations accurately. I'd say since text processing is a very important use case for LLMs, that's quite noteworthy.