4 ms·
Using tokens to evaluate models is an outdated approach. Cost per task is what matters. Not all tokens are created equal
by Bnjoroge 3mo ago
Using tokens to evaluate models is an outdated approach. Cost per task is what matters. Not all tokens are created equal
- drob518 3mo agoYes, but… more thinking tokens also means longer solution generation time. That said, v4 Flash is a fast model. I use it all the time because it’s very smart for the price. But it is verbose sometimes.
- kzrdude 3mo agoThe thinking trace was (preview) frustrating to read, I think I'd prefer a summary view of it at this point.
- WhitneyLand 3mo agoIt’s not outdated at all to use tokens to estimate performance, it’s directly related.
- prism56 3mo agoBut why should I care? If my metrics are speed and cost? How many tokens it takes as a user is arbitrary to some extent.
- WhitneyLand 3mo agoIf speed is a metric for you, tokens required to solve a problem affects that metric. All else being equal passing triple the amount of tokens through a model to solve the same problem makes it slower. Doesn’t mean this model is bad, and it has to be considered how cheap it is, but it’s a factor.
- rat9988 3mo agoStill, seems useless to measure tokens when you can measure speed.