3 ms·
Is this speculation or do you work at Anthropic? It would be cool to see the prompts used for this.
by modeless 2y ago
Is this speculation or do you work at Anthropic? It would be cool to see the prompts used for this.
- sebzim4500 2y agoWhat he is describing has become the 'standard' way to run that kind of benchmark, so he is almost certainly correct. SWE Bench [1] is the best open source benchmark. [1] https://www.swebench.com/ https://www.swebench.com/