3 ms·
Yes, how do we know Opus 4.8 hasn't been trained on the SWE-Bench examples? With a squillion dollars at stake per bench point, someone will have figured out a
by willtemperley 4mo ago
Yes, how do we know Opus 4.8 hasn't been trained on the SWE-Bench examples?
With a squillion dollars at stake per bench point, someone will have figured out a plausibly deniable way to game these benchmarks.
- stingraycharles 4mo agoEhr, the SWE bench examples are particularly horrible as those are just publicly available historical PRs. So if the models are trained on GitHub data, it will be included. So almost by design that particular benchmark is tainted, and benchmarks recall rather than reasoning.
- willtemperley 4mo agoWow that's worse than I thought, and breaks the number one rule of machine learning: you don't train the model with your test dataset.