3 ms·
Have you made that benchmarking process open so others could reproduce it? > with identical results If your results are identical, you should be very sus, som
by verdverm 9mo ago
Have you made that benchmarking process open so others could reproduce it?
> with identical results
If your results are identical, you should be very sus, something is wrong if this is true. Nothing in agentic is reliable of fully deterministic
- aSidorenkoCode 9mo agoGood benchmark results don't mean identical outputs. The task completion rate is the same: both pass the same exercises. The paths the model takes differ, but the end result is the same -> pass the tests The full benchmarking methodology and tooling will be published alongside the paper.
- verdverm 9mo agoyou used the word "identical" to describe it, not me words matter which is why I still think this is a terrible idea, I don't think it holds up in the general case and would, as a peer reviewer, be inclined to believe there is benchmark filtering that makes for good results. You should use the same benchmarks everyone else is when you write your paper
- verdverm 9mo agotimely result, https://arxiv.org/pdf/2602.12670 https://arxiv.org/pdf/2602.12670