4 ms·
Isn't the benchmark working exactly how it should in that case?
by efromvt 3mo ago
Isn't the benchmark working exactly how it should in that case?
- sillysaurusx 3mo agoI was confused too. What they’re saying is, the average task you’re likely to do if you buy the model is the main predictor of costs. So if the average task benchmark is far higher than what you’re normally doing with it, you get a skewed perspective. I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it understands the codebase.
- yreg 3mo agoNot even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.
- strbean 3mo agoRouting tasks to models by complexity like a job for a LLM. I'm sure there are degenerate cases, but I'd bet a relatively small model could do the job.
- efromvt 3mo agoExactly, so it has a success rate of 0 and infinite cost/completion on your relevant benchmark. If the benchmark doesn't map to what you need it to, then yeah, it's not a useful input.