3 ms·
Cost per defined capability. Meaning you fix the task and then find how much it cost to achieve it including reasoning, tokens etc.
by simianwords 8mo ago
Cost per defined capability. Meaning you fix the task and then find how much it cost to achieve it including reasoning, tokens etc.
- MadxX79 8mo agosource?
- simianwords 8mo agohttps://epoch.ai/data-insights/llm-inference-price-trends https://epoch.ai/data-insights/llm-inference-price-trends
- MadxX79 8mo agoFrom your link: "We excluded reasoning models from our analysis of per-token prices. Reasoning models tend to generate a much larger number of tokens than other models, making these models cost more in total to evaluate on a benchmark. This makes it misleading to compare reasoning models to other models on price per token, at a given performance level." It's just price per token. Token usage is exploding.
- simianwords 8mo agoToken usage is increasing but the link I shared is comparing price per token which is the reason they are excluding reasoning models. Reasoning models have also gotten cheaper. Generally cost to “achieve a certain task” using whatever model, even reasoning has drastically reduced. The best example is arc agi. https://arcprize.org/leaderboard https://arcprize.org/leaderboard This measures cost to achieve certain percentage of score. Fix a certain accuracy and see how much price reduces over time. It’s more than 100x.