4 ms·
How is every company able to show itself at the top of every benchmark?
by paxys 4mo ago
How is every company able to show itself at the top of every benchmark?
- adam_arthur 4mo agoNot much moat, incremental improvements, cherry picking models to compare. To be fair, seems more correct to compare against similar strength models if your main edge is pricing.
- ffsm8 4mo agoThey're being called "trust me bro benchmarks" for a reason ( ･ั ﹏ ･ั )
- logicchains 4mo agoWait to the exact moment your model is ahead on at least N benchmarks then publish.
- morgengold 4mo agoFirst look what models are worse in a set of self selected benchmarks. Second, compare to older versions of competitor s models. Still does not look good? Compare to own previous models.
- toephu2 4mo agoAnyone deep in the AI realm know which is the gold standard benchmark for coding?
- deleted 4mo ago[deleted]
- msdz 4mo agoIt’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process). “Good” benchmarks to gauge development skills at the moment seem to be: DeepSWE [0] by Datacurve FrontierCode [1] by Cognition And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore. But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again. [0] https://deepswe.datacurve.ai https://deepswe.datacurve.ai [1] https://cognition.com/blog/frontier-code https://cognition.com/blog/frontier-code [2] https://xcancel.com/i/article/2064210146558136827 https://xcancel.com/i/article/2064210146558136827
- Archit3ch 4mo agoAt this point comparing to Gemini is a free Bingo space.