3 ms·
cost per benchmark task is definitely interesting! i've always wanted cost per prompt, but even that has too much variation.
by shay_ker 3mo ago
cost per benchmark task is definitely interesting!
i've always wanted cost per prompt, but even that has too much variation.
- janilowski 3mo agoWell yeah this benchmark I've quoted is quite large, I believe they use thousands of diverse tasks, so this average cost per task could be a more accurate representation of how expensive each model actually is to run.