10 ms·

Claude 3.7 Sonnet scored 60.4% on the aider polyglot leaderboard [0], WITHOUT USING THINKING. Tied for 3rd place with o3-mini-high. Sonnet 3.7 has the highest

Claude 3.7 Sonnet scored 60.4% on the aider polyglot leaderboard [0], WITHOUT USING THINKING. Tied for 3rd place with o3-mini-high. Sonnet 3.7 has the highest non-thinking score, taking that title from Sonnet 3.5. Aider 0.75.0 is out with support for 3.7 Sonnet [1]. Thinking support and thinking benchmark results coming soon. [0] https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ [1] https://aider.chat/HISTORY.html#aider-v0750 https://aider.chat/HISTORY.html#aider-v0750

bearjaws 2y ago
Thanks for all the work on aider, my favorite AI tool.
- bt1a 2y ago
  It really is best in slot. Owe it to git, which has a particular synergy with a hallucination-prone but correctable system
  - doctoboggan 2y ago
    I like Aider but I've turned off auto-commit. I just can't seem to let the AI actually commit code for me. Do you regularly let Aider commit for you? How much do you review the code written by it?
    - sejje 2y ago
      I don't let it auto commit, either. I don't like committing in a broken state, and the llm breaks things plenty often.
      - MyOutfitIsVague 2y ago
        What's wrong with committing in a broken state if you squash those into a working state before pushing?
        sejje 2y ago
        Maybe nothing, I just don't work that way.
      - fragmede 2y ago
        The beauty of git is that local commits don't get seen by anybody until you push. so you can commit early and commit often, since no one else is gonna see it, which gets you checkpoints before, during, and after you dive into making a big breaking change in the code. once you've got something you like, then you can edit, squash, and reorder the local commits and clean them up for consumption by the general public. But to each their own!
    - bitbuilder 2y ago
      The auto-commits of Aider scared the crap out of me at first too, but after realizing I can just create a throwaway branch and let it run wild it ended up being a nice way to work. I've been trying to use Sonnet 3.7 tonight through the Copilot agent and it gets frustrating to see the API 500 halfway through the task list leaving the project in a half baked state, and then and not feeling like I have a good "auto save" to pick up again from.
    - itgoon 2y ago
      I create a feature branch, do the work and let it commit. I check the code as I go. If I don't like it, then I revert to a previous commit. Other times I write some code that it isn't getting right for whatever reason. When it's ready, I squash merge into main.
    - joshstrange 2y ago
      I originally was against auto commit as well, but now I can’t imagine not using it. It’s essentially save points along the way. More than once, I’ve done two or three exchanges with Aider only to realize that the path that we were going down was not a good one. Being able to get reset back to the last known good state is awesome. If you turn off auto commit, it’s a lot harder to undo one of the steps that the model takes. It’s only a matter of time until it creates nonsense, so you’ll really want the ability to roll it back. Just work in a branch and you can merge all commits if you want at the end.
stavros 2y ago
I'd like to second the thanks for Aider, I use it all the time.
liamYC 2y ago
I’d like to 3rd the thanks for Aider it’s fantastic!
gwd 2y ago
Interesting that the "correct diff format" score went from 99.6% with Claude 3.5 to 93.3% for Claude 3.7. My experience with using claude-code was that it consistently required several tries to get the right diff. Hopefully all that will improve as they get things ironed out.
- WatchDog 2y ago
  3.7 completed a lot more than 3.5, without seeing the actual results, we can't tell if there were any regressions in the edit format among the previously completed tasks.
- macNchz 2y ago
  Reasoning models pretty reliably seem to do worse at exacting output formats/structured outputs—so far with Aider it has been an effective strategy to employ o1 to “think” about the issue at hand, and have Sonnet implement. Interested to try various approaches with 3.7 in various combinations of reasoning effort.
  - bugglebeetle 2y ago
    It’s funny because I also have found myself doing this exact with R1+Sonnet 3.5 recently. Windsurf allows you to do a chat mode exchange with one model and then switch to another to implement. The reasoning models all seem pretty poorly implemented for the agentic workflows, but work well when paired with Claude.
- Sterling9x 2y ago
  That's a file context problem because you use cursor or cline or some other crap context maker. Try Clood. Unless "anthropic high usage" which I just watch the incident reports I one shot features regularly. At a high skill level. Not front end. Back end c# in a small but great framework that has poor documentation. Not just endpoints but full on task queues. So really, it's a context problem. You're just not laser focusing your context. Try this: Set up a context with the exact files needed. Sure ai "should" do that but it doesn't. Especially not cursor or cline. Then try. Hell try it with clood after I update with 3.7. I bet you, if you clood file it, then you get one shots. I have a long history of clood being a commit in my projects and it's a clood one shot.
throwaway454812 2y ago
Any chance you can add support for Vertex AI Sonnet 3.7, which looks like it's available now? Thank you!
deleted 2y ago
[deleted]
usaar333 2y ago
Updated. #1 with thinking
nightpool 2y ago
> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?
- anotherpaulg 2y ago
  I try not to let perfect be the enemy of good. All benchmarks have limitations. The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over the last 18+ months. To be clear, the purpose of the benchmark is to help me quantitatively assess and improve aider and make it more effective. But it's also turned out to be a great way to measure the coding skill of LLMs.
  - szundi 2y ago
    [dead]
  - jrflowers 2y ago
    >I try not to let perfect be the enemy of good. All benchmarks have limitations. Overfitting is one of the fundamental issues to contend with when trying to figure out if any type of model at all is useful. If your leaderboard corresponds to vibes and that is your target, you could just have a vibes leaderboard
  - Marazan 2y ago
    Having the verbatim answer to the test is not a "limitation" it is an invalidation.
    - deleted 2y ago
      [deleted]
  - rodrigodlu 2y ago
sheepdestroyer 2y ago
Nice ! Could we please get benchmarks for architect / DeepSeek R1 + claude-3-7-20250219 ? To compare perf and price with Sonnet-3.7-thinking
SamBam 2y ago
I like that we're just saying they're thinking now. John Searle would be furious. (I kid, I know what is meant by that.)
darkotic 2y ago
Working really well for me. Thanks for Aider!
anotherpaulg 2y ago
Using up to 32k thinking tokens, Sonnet 3.7 set SOTA with a 64.9% score. 65% Sonnet 3.7, 32k thinking 64% R1+Sonnet 3.5 62% o1 high 60% Sonnet 3.7, no thinking 60% o3-mini high 57% R1 52% Sonnet 3.5
- pclmulqdq 2y ago
  Also for $36.83 compared to o1's $186.50
  - pzo 2y ago
    But also for $36.83 compared to DeepSeek R1 + claude-3-5 it's $13.29 and for latter "Percent using correct edit format" is 100% vs 97.8% for 3.7. edit: would be interesting to see how combo DeepSeek R1 + claude-3-7 performs.
    - tw1984 2y ago
      is there any public info on why such DeepSeek R1 + claude-3-5 combo worked better than using a single model?
      - Ballas 2y ago
        From my experiments with the Deepseek Qwen-32b distill model, the Deepseek model did not follow the edit instructions - the format was wrong. I know the distill models are not at all the same as the full model, but that could provide a clue. Combine that information with the scores, then you have a reasonable hypothesis.
        re-thc 2y ago
        > I know the distill models are not at all the same as the full model It's far worse than that. It's not the model (Deepseek) at all. It's Qwen enhanced with Deepseek. So it's Qwen still.
      - alienthrowaway 2y ago
doctoboggan 2y ago
Have you tried Claude 3.7 + Deepseek as the architect? Seeing as "DeepSeek R1 + claude-3-5-sonnet-20241022" is the second place option, "DeepSeek R1 + claude-3-7" would hopefully be the highest ranking choice so far?
- SparkyMcUnicorn 2y ago
  It looks like Sonnet 3.7 (extended thinking) would be a better architect than R1. I'll be trying out Sonnet 3.7 extended thinking + Sonnet 3.5 or Flash 2.0, which I assume would be at the top of the leaderboard.
  - attentive 2y ago
    given 3.5 and 3.7 cost the same, it doesn't make sense to use 3.5 here. I'd like to see that benchmark, but R1 + 3.7 should be cheaper than 3.7T + 3.7
    - SparkyMcUnicorn 2y ago
      The reason 3.5 (as the editor) makes more sense to me is the edit format success rate (99.6% vs 3.7's 93.3%). Flash 2.0 got 100% on the edit format, and it's extremely cheap, so I'm pretty curious how that would score.
SweetSoftPillow 2y ago
r1 + Claude 3.7 when?
deleted 2y ago
[deleted]
miroljub 2y ago
And yet, "DeepSeek R1 + claude-3-5-sonnet-20241022" scores 64% on the same benchmark 30% cheaper. It's amazing what Deepseek is putting on the table while being full open source.
createaccount99 2y ago
Is aider still relevant vs. Claude Code?
- billmalarky 2y ago
  Yes. Absolutely it is. For different workloads it is an insanely effective tool.
billmalarky 2y ago
Hi Paul, been following the aider project for about a year now to develop an understanding of how to build SWE agents. I was at the AI Engineering Summit in NYC last week and met an (extremely senior) staff ai engineer doing somewhat unbelievable things with aider. Shocking things tbh. Is there a good way to share stories about real-world aider projects like this with you directly (if I can get approval from him)? Not sure posting on public forum is appropriate but I think you would be really interested to hear how people are using this tool at the edge.
- tecleandor 2y ago
  Hope it gets to be public, I love to learn "weird" (or unusual) ways of using tools

bearjaws 2y ago

Thanks for all the work on aider, my favorite AI tool.

bt1a 2y ago
It really is best in slot. Owe it to git, which has a particular synergy with a hallucination-prone but correctable system
- doctoboggan 2y ago
  I like Aider but I've turned off auto-commit. I just can't seem to let the AI actually commit code for me. Do you regularly let Aider commit for you? How much do you review the code written by it?
  - sejje 2y ago
    I don't let it auto commit, either. I don't like committing in a broken state, and the llm breaks things plenty often.
    - MyOutfitIsVague 2y ago
      What's wrong with committing in a broken state if you squash those into a working state before pushing?
      - sejje 2y ago
        Maybe nothing, I just don't work that way.
    - fragmede 2y ago
      The beauty of git is that local commits don't get seen by anybody until you push. so you can commit early and commit often, since no one else is gonna see it, which gets you checkpoints before, during, and after you dive into making a big breaking change in the code. once you've got something you like, then you can edit, squash, and reorder the local commits and clean them up for consumption by the general public. But to each their own!
  - bitbuilder 2y ago
    The auto-commits of Aider scared the crap out of me at first too, but after realizing I can just create a throwaway branch and let it run wild it ended up being a nice way to work. I've been trying to use Sonnet 3.7 tonight through the Copilot agent and it gets frustrating to see the API 500 halfway through the task list leaving the project in a half baked state, and then and not feeling like I have a good "auto save" to pick up again from.
  - itgoon 2y ago
    I create a feature branch, do the work and let it commit. I check the code as I go. If I don't like it, then I revert to a previous commit. Other times I write some code that it isn't getting right for whatever reason. When it's ready, I squash merge into main.
  - joshstrange 2y ago
    I originally was against auto commit as well, but now I can’t imagine not using it. It’s essentially save points along the way. More than once, I’ve done two or three exchanges with Aider only to realize that the path that we were going down was not a good one. Being able to get reset back to the last known good state is awesome. If you turn off auto commit, it’s a lot harder to undo one of the steps that the model takes. It’s only a matter of time until it creates nonsense, so you’ll really want the ability to roll it back. Just work in a branch and you can merge all commits if you want at the end.

stavros 2y ago

I'd like to second the thanks for Aider, I use it all the time.

liamYC 2y ago

I’d like to 3rd the thanks for Aider it’s fantastic!

gwd 2y ago

Interesting that the "correct diff format" score went from 99.6% with Claude 3.5 to 93.3% for Claude 3.7. My experience with using claude-code was that it consistently required several tries to get the right diff. Hopefully all that will improve as they get things ironed out.

WatchDog 2y ago
3.7 completed a lot more than 3.5, without seeing the actual results, we can't tell if there were any regressions in the edit format among the previously completed tasks.
macNchz 2y ago
Reasoning models pretty reliably seem to do worse at exacting output formats/structured outputs—so far with Aider it has been an effective strategy to employ o1 to “think” about the issue at hand, and have Sonnet implement. Interested to try various approaches with 3.7 in various combinations of reasoning effort.
- bugglebeetle 2y ago
  It’s funny because I also have found myself doing this exact with R1+Sonnet 3.5 recently. Windsurf allows you to do a chat mode exchange with one model and then switch to another to implement. The reasoning models all seem pretty poorly implemented for the agentic workflows, but work well when paired with Claude.
Sterling9x 2y ago
That's a file context problem because you use cursor or cline or some other crap context maker. Try Clood. Unless "anthropic high usage" which I just watch the incident reports I one shot features regularly. At a high skill level. Not front end. Back end c# in a small but great framework that has poor documentation. Not just endpoints but full on task queues. So really, it's a context problem. You're just not laser focusing your context. Try this: Set up a context with the exact files needed. Sure ai "should" do that but it doesn't. Especially not cursor or cline. Then try. Hell try it with clood after I update with 3.7. I bet you, if you clood file it, then you get one shots. I have a long history of clood being a commit in my projects and it's a clood one shot.

throwaway454812 2y ago

Any chance you can add support for Vertex AI Sonnet 3.7, which looks like it's available now? Thank you!

deleted 2y ago

[deleted]

usaar333 2y ago

Updated. #1 with thinking

nightpool 2y ago

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

anotherpaulg 2y ago
I try not to let perfect be the enemy of good. All benchmarks have limitations. The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over the last 18+ months. To be clear, the purpose of the benchmark is to help me quantitatively assess and improve aider and make it more effective. But it's also turned out to be a great way to measure the coding skill of LLMs.
- szundi 2y ago
  [dead]
- jrflowers 2y ago
  >I try not to let perfect be the enemy of good. All benchmarks have limitations. Overfitting is one of the fundamental issues to contend with when trying to figure out if any type of model at all is useful. If your leaderboard corresponds to vibes and that is your target, you could just have a vibes leaderboard
- Marazan 2y ago
  Having the verbatim answer to the test is not a "limitation" it is an invalidation.
  - deleted 2y ago
    [deleted]
- rodrigodlu 2y ago

sheepdestroyer 2y ago

Nice ! Could we please get benchmarks for architect / DeepSeek R1 + claude-3-7-20250219 ? To compare perf and price with Sonnet-3.7-thinking

SamBam 2y ago

I like that we're just saying they're thinking now. John Searle would be furious. (I kid, I know what is meant by that.)

darkotic 2y ago

Working really well for me. Thanks for Aider!

anotherpaulg 2y ago

Using up to 32k thinking tokens, Sonnet 3.7 set SOTA with a 64.9% score. 65% Sonnet 3.7, 32k thinking 64% R1+Sonnet 3.5 62% o1 high 60% Sonnet 3.7, no thinking 60% o3-mini high 57% R1 52% Sonnet 3.5

pclmulqdq 2y ago
Also for $36.83 compared to o1's $186.50
- pzo 2y ago
  But also for $36.83 compared to DeepSeek R1 + claude-3-5 it's $13.29 and for latter "Percent using correct edit format" is 100% vs 97.8% for 3.7. edit: would be interesting to see how combo DeepSeek R1 + claude-3-7 performs.
  - tw1984 2y ago
    is there any public info on why such DeepSeek R1 + claude-3-5 combo worked better than using a single model?
    - Ballas 2y ago
      From my experiments with the Deepseek Qwen-32b distill model, the Deepseek model did not follow the edit instructions - the format was wrong. I know the distill models are not at all the same as the full model, but that could provide a clue. Combine that information with the scores, then you have a reasonable hypothesis.
      - re-thc 2y ago
        > I know the distill models are not at all the same as the full model It's far worse than that. It's not the model (Deepseek) at all. It's Qwen enhanced with Deepseek. So it's Qwen still.
    - alienthrowaway 2y ago

doctoboggan 2y ago

Have you tried Claude 3.7 + Deepseek as the architect? Seeing as "DeepSeek R1 + claude-3-5-sonnet-20241022" is the second place option, "DeepSeek R1 + claude-3-7" would hopefully be the highest ranking choice so far?

SparkyMcUnicorn 2y ago
It looks like Sonnet 3.7 (extended thinking) would be a better architect than R1. I'll be trying out Sonnet 3.7 extended thinking + Sonnet 3.5 or Flash 2.0, which I assume would be at the top of the leaderboard.
- attentive 2y ago
  given 3.5 and 3.7 cost the same, it doesn't make sense to use 3.5 here. I'd like to see that benchmark, but R1 + 3.7 should be cheaper than 3.7T + 3.7
  - SparkyMcUnicorn 2y ago
    The reason 3.5 (as the editor) makes more sense to me is the edit format success rate (99.6% vs 3.7's 93.3%). Flash 2.0 got 100% on the edit format, and it's extremely cheap, so I'm pretty curious how that would score.

SweetSoftPillow 2y ago

r1 + Claude 3.7 when?

deleted 2y ago

[deleted]

miroljub 2y ago

And yet, "DeepSeek R1 + claude-3-5-sonnet-20241022" scores 64% on the same benchmark 30% cheaper. It's amazing what Deepseek is putting on the table while being full open source.

createaccount99 2y ago

Is aider still relevant vs. Claude Code?

billmalarky 2y ago
Yes. Absolutely it is. For different workloads it is an insanely effective tool.

billmalarky 2y ago

Hi Paul, been following the aider project for about a year now to develop an understanding of how to build SWE agents. I was at the AI Engineering Summit in NYC last week and met an (extremely senior) staff ai engineer doing somewhat unbelievable things with aider. Shocking things tbh. Is there a good way to share stories about real-world aider projects like this with you directly (if I can get approval from him)? Not sure posting on public forum is appropriate but I think you would be really interested to hear how people are using this tool at the edge.

tecleandor 2y ago
Hope it gets to be public, I love to learn "weird" (or unusual) ways of using tools