3 ms·
The link you are commenting on shows data from actual prompts from real users, and the COST of the average prompt increased 37%. I do not think synthetic benchm
by irthomasthomas 6mo ago
The link you are commenting on shows data from actual prompts from real users, and the COST of the average prompt increased 37%. I do not think synthetic benchmarks are a rebuttal to real usage data.
- andai 6mo agoThe cost of the input tokens, not the reasoning or output. Agree though that benchmarks aren't very helpful w.r.t. estimating real world performance or costs. What we'd need are people giving the same real world tasks to 4.6 and 4.7 and measuring time, quality and costs.
- irthomasthomas 6mo agoThanks, that wasn't clear because it mentioned conversations, but it is only measuring the input tokens. So its just measuring the difference in the tokenizer.