3 ms·
This is what Anthropic does for their "High Compute" SWE benchmarking: " For our "high compute" numbers we adopt additional complexity and parallel test-time c
by datadrivenangel 1y ago
This is what Anthropic does for their "High Compute" SWE benchmarking:
"
For our "high compute" numbers we adopt additional complexity and parallel test-time compute as follows:
- We sample multiple parallel attempts.
- We discard patches that break the visible regression tests in the repository, similar to the rejection sampling approach adopted by Agentless (Xia et al. 2024); note no hidden test information is used.
- We then use an internal scoring model to select the best candidate from the remaining attempts.
- This results in a score of 82.0% for Sonnet 4.5.
"
But running in parallel means you use more compute, meaning that the cost is higher. Good results are worth paying for, but if a super recursive approach costs more and takes longer than a human...