3 ms·
Shameless self plug but also worried about the silent quality regressions, I started building a tool to track coding agent performance over time.. https://gith
by siliconc0w 6mo ago
Shameless self plug but also worried about the silent quality regressions, I started building a tool to track coding agent performance over time.. https://github.com/s1liconcow/repogauge https://github.com/s1liconcow/repogauge
Here is a sample report that tries out the cheaper models + the newest Kimi2.6 model against the 5.4 'gold' testcases from the repo: https://repogauge.org/sample_report https://repogauge.org/sample_report.
- conception 6mo agoThis is cool - just wanted to note https://marginlab.ai https://marginlab.ai is one that has been around for a while.
- aleksiy123 6mo agoare there any tools anyone knows to collect this kind of telemetry while using the tools instead of offline evals. running evals seems like it may be a bit too expensive as a solo dev.