11 ms·
Without benchmarks and/or a whole suite of non-cherrypicked examples, this means nothing because you can trivially make an AI generate anything from text.
by amelius 6mo ago
Without benchmarks and/or a whole suite of non-cherrypicked examples, this means nothing because you can trivially make an AI generate anything from text.
- softservo 6mo agoWorking on benchmarks at the moment! Always open to feedback / PRs.
- carterschonwald 6mo agoim def working on benchmarks for how my own general harness improves task performance vs same model in a commodity setup. its hard to do! i will say that my current harness: https://github.com/cartazio/oh-punkin-pi https://github.com/cartazio/oh-punkin-pi is a testbed for a bunch of 2nd gen harness tech, largely optimized for reasoning llms only. the next one after this harness is gonna be epicccc