3 ms·
Wouldn’t solutions be in the training data? The game is quite old.
by dash2 3mo ago
Wouldn’t solutions be in the training data? The game is quite old.
- stared 3mo agoSee the "caveats" section. It was also our initial assumption, but to our surprise there were no signs of models "knowing" the solutions (we investigated all trajectories). Compare and contrast with SWE-Bench Verified, for which models suddenly generate solutions, or at least, have substantial hints, https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ https://openai.com/index/why-we-no-longer-evaluate-swe-bench....