> The gap here is really two different definitions of "correct." Tests verify behavior. Reviewers verify fit Take a second look at the "reasons for rejection".

> The gap here is really two different definitions of "correct." Tests verify behavior. Reviewers verify fit Take a second look at the "reasons for rejection". A quarter or so of the changes that pass tests actually don't solve the problem they intend to. The tests used in SWE-bench do not discriminate all that well between working and broken solutions.