4 ms·
We recently added cost (last week), so data is sparse. Check back in a few weeks and it will be represented somewhere on the homepage, probably in the Efficienc
by gertlabs 6mo ago
We recently added cost (last week), so data is sparse. Check back in a few weeks and it will be represented somewhere on the homepage, probably in the Efficiency Chart at the bottom. We also plan to show model performance deviation over time after we collect more data.
I'm interested to hear about any other data representations you'd like to see, too. The goal is to convey the most important information as densely as possible, without too much clutter.
- DeathArrow 6mo ago>I'm interested to hear about any other data representations you'd like to see, too It would be nice if you can show how much the models drift from the instructions over time
- gertlabs 6mo agoNot sure what you mean. Time series chart of model performance over time to see if proprietary models get degraded? That's in the works, but we will need a couple months more data collection before launch.
- DeathArrow 6mo agoYes, probably performance helps. The idea is that the larger a coding task is and the longer the coding agent is, the higher the chance is for the agent to not follow the rules and guidelines.