4 ms·
Good benchmark results don't mean identical outputs. The task completion rate is the same: both pass the same exercises. The paths the model takes differ, but t
by aSidorenkoCode 8mo ago
Good benchmark results don't mean identical outputs. The task completion rate is the same: both pass the same exercises. The paths the model takes differ, but the end result is the same -> pass the tests
The full benchmarking methodology and tooling will be published alongside the paper.
- verdverm 8mo agoyou used the word "identical" to describe it, not me words matter which is why I still think this is a terrible idea, I don't think it holds up in the general case and would, as a peer reviewer, be inclined to believe there is benchmark filtering that makes for good results. You should use the same benchmarks everyone else is when you write your paper
- verdverm 8mo agotimely result, https://arxiv.org/pdf/2602.12670 https://arxiv.org/pdf/2602.12670