4 ms·
I don't get the point. The model has presumably been trained on all public GitHub code, so the evaluation is tainted anyway.
by fsh 5mo ago
I don't get the point. The model has presumably been trained on all public GitHub code, so the evaluation is tainted anyway.
- adrian_b 5mo agoA couple of days ago there has been another thread about an experiment with many LLMs, where especially the Anthropic models were found to "cheat" in a large percentage of the coding tasks that had been benchmarked, by searching the Internet for appropriate code and inserting it in the program they had to write. The conclusion of that study was that when benchmarking LLMs for coding ability, they should not have access to Internet, if you want to know their intrinsic abilities. Moreover, this can be worrisome as a more direct copyright infringement than the one caused by training, because even if they find open source code on the Internet and they insert it in the generated files, it is pretty certain that it must have had a license that prohibits the removal of the copyright notice.
- htrp 5mo ago> A couple of days ago there has been another thread about an experiment with many LLMs, where especially the Anthropic models were found to "cheat" in a large percentage of the coding tasks that had been benchmarked, by searching the Internet for appropriate code and inserting it in the program they had to write. Can you find the thread?
- adrian_b 5mo agoI have found it: https://news.ycombinator.com/item?id=48045174 https://news.ycombinator.com/item?id=48045174 The study paper: https://arxiv.org/abs/2605.03546 https://arxiv.org/abs/2605.03546 Look at Table 3, where the cheating rates of Claude Sonnet, Claude Opus and Gemini were between 20% and 36%, during the coding benchmarks.
- ej88 5mo agoswe bench pro has a public and private test set, where the private eval is from proprietary codebases only