3 ms·
>I'm interested to hear about any other data representations you'd like to see, too It would be nice if you can show how much the models drift from the instruc
by DeathArrow 6mo ago
>I'm interested to hear about any other data representations you'd like to see, too
It would be nice if you can show how much the models drift from the instructions over time
- gertlabs 6mo agoNot sure what you mean. Time series chart of model performance over time to see if proprietary models get degraded? That's in the works, but we will need a couple months more data collection before launch.
- DeathArrow 6mo agoYes, probably performance helps. The idea is that the larger a coding task is and the longer the coding agent is, the higher the chance is for the agent to not follow the rules and guidelines.