4 ms·

claude swe-bench is 80.8 and codex is 56.8 Seems like 4.6 is still all-around better?

• gizmodo59 8mo ago
  Its SWE bench pro not swe bench verified. The verified benchmark has stagnated
  • joshuahedlund 8mo ago
    Any ideas why verified has stagnated? It was increasing rapidly and then basically stopped.
    • Snuggly73 8mo ago
      it has been pretty much a benchmark for memorization for a while. there is a paper on the subject somewhere. swe bench pro public is newer, but its not live, so it will get slowly memorized as well. the private dataset is more interesting, as are the results there: https://scale.com/leaderboard/swe_bench_pro_private https://scale.com/leaderboard/swe_bench_pro_private
• Rudybega 8mo ago
  You're comparing two different benchmarks. Pro vs Verified.