2 ms·
There are plenty. But it's not the comparison you want to be making. There is too much variability between the number of tokens used for a single response, espe
by jsnell 10mo ago
There are plenty. But it's not the comparison you want to be making. There is too much variability between the number of tokens used for a single response, especially once reasoning models became a thing. And it gets even worse when you put the models into a variable length output loop.
You really need to look at the cost per task. artificialanalysis.ai has a good composite score, measures the cost of running all the benchmarks, and has 2d a intelligence vs. cost graph.
- misiti3780 10mo agothanks
- deaux 10mo agoFor reference the above completely depends on what you're using them for. For many tasks, the number of tokens used is consistent within 10~20%.