What he is describing has become the 'standard' way to run that kind of benchmark, so he is almost certainly correct. SWE Bench [1] is the best open source benc

What he is describing has become the 'standard' way to run that kind of benchmark, so he is almost certainly correct. SWE Bench [1] is the best open source benchmark. [1] https://www.swebench.com/ https://www.swebench.com/