2 ms·

The SWEBench benchmarks are really gamed at this point and should not be trusted period. The solutions are effectively in the training sets and have been for a

The SWEBench benchmarks are really gamed at this point and should not be trusted period. The solutions are effectively in the training sets and have been for a while.

mnicky 3mo ago
SWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.
- saberience 3mo ago
  SWE Bench Pro is also gamed and shouldn't be trusted.
  - mnicky 3mo ago
    Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.
  - Sabinus 3mo ago
    Why that benchmark in particular?