4 ms·
Somehow regresses on SWE bench?
by osti 8mo ago
Somehow regresses on SWE bench?
- usaar333 8mo agoi'd interpret that as rounding error. that is unchanged swe-bench seems really hard once you are above 80%
- Squarex 8mo agoit's not a great benchmark anymore... starting with it being python / django primarily... the industry should move to something more representative
- usaar333 8mo agoOpenai has; they don't even mention score on gpt-5.3-codex. On the other hand, it is their own verified benchmark, which is telling.
- lkbm 8mo agoI don't know how these benchmarks work (do you do a hundred runs? A thousand runs?), but 0.1% seems like noise.
- SubiculumCode 8mo agoThat benchmark is pretty saturated, tbh. A "regression" of such small magnitude could mean many different things or nothing at all.