3 ms·
Not even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be
by yreg 3mo ago
Not even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.
- strbean 3mo agoRouting tasks to models by complexity like a job for a LLM. I'm sure there are degenerate cases, but I'd bet a relatively small model could do the job.
- efromvt 3mo agoExactly, so it has a success rate of 0 and infinite cost/completion on your relevant benchmark. If the benchmark doesn't map to what you need it to, then yeah, it's not a useful input.