4 ms·

For Context: SWE-Bench (+ Verified) is the benchmark (of resolving Github Issues) that companies into Coding are chasing - Devin, Claude, OpenAI - all these!

by amrrs 2y ago

For Context: SWE-Bench (+ Verified) is the benchmark (of resolving Github Issues) that companies into Coding are chasing - Devin, Claude, OpenAI - all these! A new leader #1 - CodeStory Midwit Agent + swe-search - has been crowed with a score of 62% on SWE-bench verified (without even using any reasoning models like OpenAI o1 or o3) More details on their approach - https://aide.dev/blog/sota-bitter-lesson https://aide.dev/blog/sota-bitter-lesson

alach11 2y ago
This is a very impressive result. OpenAI was able to achieve 72% with o3, but that's at a very high compute cost at inference-time. I'd be interested for Aide to release more metrics on token counts, total expenditure, etc. to better understand exactly how much test-time compute is involved here. They allude to it being a lot, but it would be nice to compare with OpenAI's o3.
- amrrs 2y ago
  tbh there has been some issue with their previous reporting https://x.com/Alex_Cuadron/status/1876017241042587964 https://x.com/Alex_Cuadron/status/1876017241042587964
- skp1995 2y ago
  Hey! One of the creators of Aide here. ngl the total expenditure was around $10k, in terms of test-time compute we ran upto 20X agents on the same problem to first understand if the bitter lesson paradigm of "scale is the answer" really holds true. The final submission which we did ran 5X agents and the decider was based on mean average score of the rewards, per problem the cost was around $20 We are going to push this scaling paradigm a bit more, my honest gut feeling is that swe-bench as a benchmark is prime for saturation real soon 1. These problem statements are in the training data for the LLMs 2. Brute-forcing the answer the way we are doing works and we just proved it, so someone is going to take a better stab at it real soon
dang 2y ago
Thanks! It feels like we should switch the top link to that URL since it's a deeper dive into the new bit that's interesting here. Edit: I've done that now. Submitted URL was https://www.swebench.com/ https://www.swebench.com/ and submitted title was "SWE Bench just got updated – new #1s".

by amrrs 2y ago

alach11 2y ago

This is a very impressive result. OpenAI was able to achieve 72% with o3, but that's at a very high compute cost at inference-time. I'd be interested for Aide to release more metrics on token counts, total expenditure, etc. to better understand exactly how much test-time compute is involved here. They allude to it being a lot, but it would be nice to compare with OpenAI's o3.

amrrs 2y ago
tbh there has been some issue with their previous reporting https://x.com/Alex_Cuadron/status/1876017241042587964 https://x.com/Alex_Cuadron/status/1876017241042587964
skp1995 2y ago
Hey! One of the creators of Aide here. ngl the total expenditure was around $10k, in terms of test-time compute we ran upto 20X agents on the same problem to first understand if the bitter lesson paradigm of "scale is the answer" really holds true. The final submission which we did ran 5X agents and the decider was based on mean average score of the rewards, per problem the cost was around $20 We are going to push this scaling paradigm a bit more, my honest gut feeling is that swe-bench as a benchmark is prime for saturation real soon 1. These problem statements are in the training data for the LLMs 2. Brute-forcing the answer the way we are doing works and we just proved it, so someone is going to take a better stab at it real soon

dang 2y ago

Thanks! It feels like we should switch the top link to that URL since it's a deeper dive into the new bit that's interesting here. Edit: I've done that now. Submitted URL was https://www.swebench.com/ https://www.swebench.com/ and submitted title was "SWE Bench just got updated – new #1s".