3 ms·

> inference costs will clearly drop They haven't though. On two fronts: 1, the soa models have been pretty constantly priced, and everyone wants the soa models

by x0x0 1y ago

> inference costs will clearly drop They haven't though. On two fronts: 1, the soa models have been pretty constantly priced, and everyone wants the soa models. Likely the only way costs drop is the models get so good that people are like hey, I'm fine with a less useful answer (which is still good enough) and that seems, right now, like a bad bet. and 2 - we use a lot more tokens now. No more pasting Q&A into a site; now people hammer up chunks of their codebases and would love to push more. More context, more thinking, more everything.

ctoth 1y ago
You're describing increased spending while calling it increased cost. These aren't the same thing. A task that cost me $5 to accomplish with GPT-4 last year might cost $1 with Sonnet today, even though I'm now spending $100/month total on AI instead of $20 because I'm doing 100x more tasks. The cost per task dropped 80%. My spending went up 5x. Both statements are true. Here's an analogy you may understand: https://crespo.business/posts/cost-of-inference/ https://crespo.business/posts/cost-of-inference/
- KallDrexx 1y ago
  Fwiw that's not necessarily true, because if Sonnet ends up using reasoning, then you are using more tokens than GPT-4 would have used for the same task. Same with GPT-5 since it will decide (using an LLM) if it should use the thinking model for it (and you don't have as much control over it).
  - steveklabnik 1y ago
    This is addressed in the post.
    - x0x0 1y ago
      Right, but if I understand you, the counterargument is dumb since the context in which we are discussing is business viability (vcs investing in businesses where the unit economics require inference cost decreases), so actual dollars out rather than some imaginary cost per token is the metric that matters. Inference is getting so much cheaper that cursor and zed have had to raise prices.
      - steveklabnik 1y ago
        > so actual dollars out rather than some imaginary cost per token is the metric that matters. Even if we take this as true, the point is that this is different than "the cost of inference isn't going down." It is going down, it's just that people want more performance, and are willing to pay for it. Spend going up is not the same as cost going up. I don't disagree that there are a wide variety of things to talk about here, but that means it's extra important to get what you're talking about straight.
        x0x0 1y ago
        Playing word games labeling inference narrowly as the cost per token rather than the per-X $ going to your llm api provider per customer/user/use/whatever is kinda silly? The cost of inference -- ie $ that go to your llm api provider -- has increased and certainly appears to continue to increase. see also https://ethanding.substack.com/p/ai-subscriptions-get-short-squeezed https://ethanding.substack.com/p/ai-subscriptions-get-short-...
        steveklabnik 1y ago
        > The cost of inference -- ie $ that go to your llm api provider This is the crux of it: when talking about "the cost of inference" for the purposes of the unit economics of the business, what's being discussed is not what they charge you. It's about their COGs. That's not word games. It's about being clear about what's being talked about. Talking about increased prices is something that could be talked about! But it's a different thing. For example, what you're talking about here is total spend, not about individual pricing going up or down. That's also a third thing! You can't come to agreement unless you agree on what's being discussed.
      - dcre 1y ago
        Why do the unit economics require a decrease in inference spend per user? This is discussed at the end of the post. I think this is based on the very strange assumption that these businesses must charge $20 a month no matter how much inference their customers want to use. This is precisely what the move to usage-based pricing was about. End users want to use more inference because they like it so much, and are knocking down these companies’ doors demanding to be allowed to pay them more money to get more inference.
infecto 1y ago
Anecdote of 1. Costs for openai on a per token basis have absolutely dropped and that accounts for new sota models over time. I think by now we can all agree that inference costs from providers are largely at or above breakeven. So more tokens is a good problem to have.
dexwiz 1y ago
Point 2 was the analysis I saw. Context size and token cost grow inversely at a rate that keeps prices constant, almost like supply and demand curves.

by x0x0 1y ago

ctoth 1y ago

You're describing increased spending while calling it increased cost. These aren't the same thing. A task that cost me $5 to accomplish with GPT-4 last year might cost $1 with Sonnet today, even though I'm now spending $100/month total on AI instead of $20 because I'm doing 100x more tasks. The cost per task dropped 80%. My spending went up 5x. Both statements are true. Here's an analogy you may understand: https://crespo.business/posts/cost-of-inference/ https://crespo.business/posts/cost-of-inference/

KallDrexx 1y ago
Fwiw that's not necessarily true, because if Sonnet ends up using reasoning, then you are using more tokens than GPT-4 would have used for the same task. Same with GPT-5 since it will decide (using an LLM) if it should use the thinking model for it (and you don't have as much control over it).
- steveklabnik 1y ago
  This is addressed in the post.
  - x0x0 1y ago
    Right, but if I understand you, the counterargument is dumb since the context in which we are discussing is business viability (vcs investing in businesses where the unit economics require inference cost decreases), so actual dollars out rather than some imaginary cost per token is the metric that matters. Inference is getting so much cheaper that cursor and zed have had to raise prices.
    - steveklabnik 1y ago
      > so actual dollars out rather than some imaginary cost per token is the metric that matters. Even if we take this as true, the point is that this is different than "the cost of inference isn't going down." It is going down, it's just that people want more performance, and are willing to pay for it. Spend going up is not the same as cost going up. I don't disagree that there are a wide variety of things to talk about here, but that means it's extra important to get what you're talking about straight.
      - x0x0 1y ago
        Playing word games labeling inference narrowly as the cost per token rather than the per-X $ going to your llm api provider per customer/user/use/whatever is kinda silly? The cost of inference -- ie $ that go to your llm api provider -- has increased and certainly appears to continue to increase. see also https://ethanding.substack.com/p/ai-subscriptions-get-short-squeezed https://ethanding.substack.com/p/ai-subscriptions-get-short-...
        steveklabnik 1y ago
        > The cost of inference -- ie $ that go to your llm api provider This is the crux of it: when talking about "the cost of inference" for the purposes of the unit economics of the business, what's being discussed is not what they charge you. It's about their COGs. That's not word games. It's about being clear about what's being talked about. Talking about increased prices is something that could be talked about! But it's a different thing. For example, what you're talking about here is total spend, not about individual pricing going up or down. That's also a third thing! You can't come to agreement unless you agree on what's being discussed.
    - dcre 1y ago
      Why do the unit economics require a decrease in inference spend per user? This is discussed at the end of the post. I think this is based on the very strange assumption that these businesses must charge $20 a month no matter how much inference their customers want to use. This is precisely what the move to usage-based pricing was about. End users want to use more inference because they like it so much, and are knocking down these companies’ doors demanding to be allowed to pay them more money to get more inference.

infecto 1y ago

Anecdote of 1. Costs for openai on a per token basis have absolutely dropped and that accounts for new sota models over time. I think by now we can all agree that inference costs from providers are largely at or above breakeven. So more tokens is a good problem to have.

dexwiz 1y ago

Point 2 was the analysis I saw. Context size and token cost grow inversely at a rate that keeps prices constant, almost like supply and demand curves.