3 ms·
Are we testing the model or the harness? If benchmarks show certain numbers it should perform as such without it
by lousken 2mo ago
Are we testing the model or the harness? If benchmarks show certain numbers it should perform as such without it
- deleted 2mo ago[deleted]
- ApolloFortyNine 2mo agoYou can't even run a benchmark without a basic harness, of course the harness has some effect.
- 0xbadcafebee 2mo agoYep. It's a state machine; change the state, change the result. https://arxiv.org/html/2605.23950v1 https://arxiv.org/html/2605.23950v1 | https://arxiv.org/pdf/2505.15146 https://arxiv.org/pdf/2505.15146 | https://medium.com/@amontzamir/youre-praising-the-wrong-thing-c14028e2459e https://medium.com/@amontzamir/youre-praising-the-wrong-thin...