3 ms·

SWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.

• saberience 3mo ago
  SWE Bench Pro is also gamed and shouldn't be trusted.
  • mnicky 3mo ago
    Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.
  • Sabinus 3mo ago
    Why that benchmark in particular?