2 ms·
It isn’t allowed to see the final evaluation test (used in calculating its pass/fail), but it can run code and see the output of its own code in order to unders
by svnt 2y ago
It isn’t allowed to see the final evaluation test (used in calculating its pass/fail), but it can run code and see the output of its own code in order to understand what doesn’t work. If it ends up creating tests as part of that based on the original problem statement then presumably that’s allowed.
- modeless 2y agoIs this speculation or do you work at Anthropic? It would be cool to see the prompts used for this.
- sebzim4500 2y agoWhat he is describing has become the 'standard' way to run that kind of benchmark, so he is almost certainly correct. SWE Bench [1] is the best open source benchmark. [1] https://www.swebench.com/ https://www.swebench.com/