4 ms·
> Run the same agent n times to increase success rate. Are there benchmarks out there that back this claim?
by hmokiguess 7mo ago
> Run the same agent n times to increase success rate.
Are there benchmarks out there that back this claim?
- danoandco 7mo agoYes, this is the pass@k metric from code generation research. Found the relevant paper Evaluating Large Language Models Trained on Code (Chen et al., 2021) which introduced the metric.
- hmokiguess 7mo agoInteresting, and how does Twill uses it in that feature?
- danoandco 7mo ago[dead]
- danoandco 7mo agoOn the Twill web app, you can run the same task across different agents and multiple attempts (each in its own sandbox). Then you pick the best result. This is super handy for UI work where you can open the live preview for each attempt and compare. Next step for us is adding a final pass where an agent evaluates the results and combines the best parts into one PR.
- j_gonzalez 7mo ago[flagged]
- lmeyerov 7mo agoWe find it true in Louie.ai evals (ai for investigations), about a 10-20% lift which meaningful. It'd measured here: botsbench.com . Unfortunately, undesirable in practice due to people being token-constrained even before. One case is retrying only on failure, but even that is a bit tricky...