4 ms·

From the Anthropic model guide: Agentic Coding Claude 3.5 Sonnet solves 64% of problems on an internal agentic coding evaluation, compared to 38% for Claude 3

From the Anthropic model guide: Agentic Coding Claude 3.5 Sonnet solves 64% of problems on an internal agentic coding evaluation, compared to 38% for Claude 3 Opus. Our evaluation tests a model’s ability to understand an open source codebase and implement a pull request, such as a bug fix or new feature, given a natural language description of the desired improvement. For each problem, the model is evaluated based on whether all the tests of the codebase pass for the completed code submission. The tests are not visible to the model, and include tests of the bug fix or new feature. To ensure the evaluation mimics real world software engineering, we based the problems on real pull requests submitted to open source codebases. The changes involve searching, viewing, and editing multiple files (typically three or four, as many as twenty). The model is allowed to write and run code in an agentic loop and iteratively self-correct during evaluation. We run these tests in a secure sandboxed environment without access to the internet. % of problems which pass all tests Claude 3.5 Sonnet 64% Claude 3 Opus 38% Claude 3 Sonnet 21% Claude 3 Haiku 17%

modeless 2y ago
> The tests are not visible to the model > The model is allowed to write and run code in an agentic loop and iteratively self-correct during evaluation What does this mean? How does the model iteratively self-correct without seeing the tests? Can it see the test results?
- svnt 2y ago
  It isn’t allowed to see the final evaluation test (used in calculating its pass/fail), but it can run code and see the output of its own code in order to understand what doesn’t work. If it ends up creating tests as part of that based on the original problem statement then presumably that’s allowed.
  - modeless 2y ago
    Is this speculation or do you work at Anthropic? It would be cool to see the prompts used for this.
    - sebzim4500 2y ago
      What he is describing has become the 'standard' way to run that kind of benchmark, so he is almost certainly correct. SWE Bench [1] is the best open source benchmark. [1] https://www.swebench.com/ https://www.swebench.com/

modeless 2y ago

> The tests are not visible to the model > The model is allowed to write and run code in an agentic loop and iteratively self-correct during evaluation What does this mean? How does the model iteratively self-correct without seeing the tests? Can it see the test results?

svnt 2y ago
It isn’t allowed to see the final evaluation test (used in calculating its pass/fail), but it can run code and see the output of its own code in order to understand what doesn’t work. If it ends up creating tests as part of that based on the original problem statement then presumably that’s allowed.
- modeless 2y ago
  Is this speculation or do you work at Anthropic? It would be cool to see the prompts used for this.
  - sebzim4500 2y ago
    What he is describing has become the 'standard' way to run that kind of benchmark, so he is almost certainly correct. SWE Bench [1] is the best open source benchmark. [1] https://www.swebench.com/ https://www.swebench.com/