4 ms·
This is an interesting idea, without giving away your benchmarks specifically what kind of stuff do you test? I might try to assemble something myself, it's so
by matt2000 2mo ago
This is an interesting idea, without giving away your benchmarks specifically what kind of stuff do you test? I might try to assemble something myself, it's so hard to determine model quality from the system card these days.
- eli 2mo agoI have actually been planning to open source the framework. Only benchmarks I care about are the ones that look like real work I do. So it makes it easy to trawl your own repos looking for benchmark task candidates from real bug fixes or features. Then as a bonus I can compare the agent’s solution to my own as a reference. So most look like that but I did include a few one-shot “build an app that solves this problem” and some qualitative design tasks and a tough algorithmic optimization one.
- matt2000 2mo agoWould open sourcing it make you feel like the results might be compromised? I'd rather keep it private so it's specific to my use cases and not included in any training date (no matter now small the signal in the overall data).
- eli 2mo agoOh sorry I meant the framework. Though I could see posting a couple of the tasks.
- bisonbear 2mo agoAlso working on a product to build tasks from your own work for testing coding agents. Main thing I would offer is to look carefully at the agent trajectories - they love to figure out ways to cheat. Additionally, consider what "winning" means. If just using test pass rate, consider that tests might not encode what good means in your repo. I have been having success using "equivalence with merged PR" as judged by an LLM as a signal.
- eli 2mo agoYup, trying to be really strategic about testing. I didn't end up sticking with it, but I tried requiring test cases to cite a matching clause in the task assignment. But also: these tests are only indicative. Only a human can score a run.