4 ms·
Considering the DeepSWE result (imho if you're gonna give value to benchmarks this is one of the best) it's not good enough.
by lnenad 3mo ago
Considering the DeepSWE result (imho if you're gonna give value to benchmarks this is one of the best) it's not good enough.
- svantana 3mo agoIt's a high quality benchmark for sure, but it being public means it's at risk of leaking into the models (unintentionally or not), right? For that reason I prefer to look at the private ones, like: HLE, SimpleBench, Kagi, ARC-AGI.