3 ms·
Not sure what you mean by benchmaxxing but we think there's still a lot of useful signals you can infer from SWE-bench-style benchmarking. We also have SWE-ben
by ofirpress 1y ago
Not sure what you mean by benchmaxxing but we think there's still a lot of useful signals you can infer from SWE-bench-style benchmarking.
We also have SWE-bench Multimodal which adds a twist I haven't seen elsewhere:
https://www.swebench.com/multimodal.html https://www.swebench.com/multimodal.html
- Snuggly73 1y agoI mean that there is the possibility that swe bench is being specifically targeted for training and the results may not reflect real world performance.