4 ms·

I think this is a minor speed bump and VC’s believe that cost of inference will decrease over time and this is a gold rush to grab market share while cost of in

I think this is a minor speed bump and VC’s believe that cost of inference will decrease over time and this is a gold rush to grab market share while cost of inference declines. I don’t think they got it right and the market share and usage grew faster than inference dropped, but inference costs will clearly drop and these companies will eventually be very profitable. Reality is that startups like this assume moore’s law will drop the cost over time and arrange their business around where they expect costs to be and not where costs currently are.

Frieren 1y ago
> I think this is a minor speed bump and VC’s believe that cost of inference will decrease over time and this is a gold rush to grab market share while cost of inference declines. It could also be that you give too much credit to the market. People follow trends because in most cases that makes money. There is no other deeper though involved. Look at the financial crisis, totally irrational.
- rglover 1y ago
  This. Post-crypto, AI was the obvious next gambit for VC. Their money flows to hype, not product. The second that hype starts to fade and the money dries up, VCs will be running with their Harold Hill trunk full of cash toward the border. Just from the content they publish alone you can tell they're channeling their inner Barnum & Bailey in between Ayahuasca seizures.
xnx 1y ago
> inference costs will clearly drop and these companies will eventually be very profitable. Inference costs for old models will drop, but inference costs may stay the same if models continue to improve. No guarantee that any wrapper for inference will be able to hold on to customers when they stop selling $1.00 for $0.50.
- deleted 1y ago
  [deleted]
onlyrealcuzzo 1y ago
Isn't the consensus that the MOE architecture and other optimizations in the newest gen models (GPT-5, Gemini 3.0 to come, etc) will reduce inference costs by 50-75% already?
- yomismoaqui 1y ago
  Sounds interesting, do you have some links with more info about this? Thanks!
- ACCount37 1y ago
  Kind of. Frontier LLMs aren't going to get cheaper, but that's because the frontier keeps advancing. Price-performance though? The trend is clear: a given level of LLM capability keeps getting cheaper, and that trend is expected to hold. Improvements in architecture and training make LLMs more capability-dense, and advanced techniques make inference cheaper.
  - onlyrealcuzzo 1y ago
    > Frontier LLMs aren't going to get cheaper One of the main selling points to MOE, is that the architecture is designed such that you can re-train experts independently, as well as add new experts, change the size of an experts parameters, etc, without retraining the entire model. If 80% of you usage comes from 20% of your experts, you can cut your future training costs SUBSTANTIALLY.
    - ACCount37 1y ago
      I don't think anyone has ever managed to pull off a success story with this kind of advanced expert manipulation. It's not entirely impossible, but I remain skeptical until see a proof that it, first, works. And, second, that it actually has an advantage over "we'll just train another base model from scratch, but 10% larger, with those +5% performance architecture tweaks, and a new modality blender, and more of that good highly curated data in the dataset, and fresher data overall, and it'll be glorious".
x0x0 1y ago
> inference costs will clearly drop They haven't though. On two fronts: 1, the soa models have been pretty constantly priced, and everyone wants the soa models. Likely the only way costs drop is the models get so good that people are like hey, I'm fine with a less useful answer (which is still good enough) and that seems, right now, like a bad bet. and 2 - we use a lot more tokens now. No more pasting Q&A into a site; now people hammer up chunks of their codebases and would love to push more. More context, more thinking, more everything.
- ctoth 1y ago
  You're describing increased spending while calling it increased cost. These aren't the same thing. A task that cost me $5 to accomplish with GPT-4 last year might cost $1 with Sonnet today, even though I'm now spending $100/month total on AI instead of $20 because I'm doing 100x more tasks. The cost per task dropped 80%. My spending went up 5x. Both statements are true. Here's an analogy you may understand: https://crespo.business/posts/cost-of-inference/ https://crespo.business/posts/cost-of-inference/
  - KallDrexx 1y ago
    Fwiw that's not necessarily true, because if Sonnet ends up using reasoning, then you are using more tokens than GPT-4 would have used for the same task. Same with GPT-5 since it will decide (using an LLM) if it should use the thinking model for it (and you don't have as much control over it).
    - steveklabnik 1y ago
      This is addressed in the post.
      - x0x0 1y ago
        Right, but if I understand you, the counterargument is dumb since the context in which we are discussing is business viability (vcs investing in businesses where the unit economics require inference cost decreases), so actual dollars out rather than some imaginary cost per token is the metric that matters. Inference is getting so much cheaper that cursor and zed have had to raise prices.
Analemma_ 1y ago
My own usage and the usage of pretty much everyone I know says that as inference costs drop, usage goes up in lockstep, and I’m still nowhere near the ceiling of how many tokens I could use if they were free. I think if these companies are gambling their future on COGS going down, that’s a gamble they’re going to lose.
- delaminator 1y ago
  aka Jevons Paradox. we strive to make machines more efficient to reduce energy usage. That makes $ per unit of work lower. But instead of reducing spending, people find more uses for the machines.
username223 1y ago
Color me skeptical. We're running into the speed of light when it comes to transistor size, and the parallelism that made neural nets take off is running into power demands. Where do the exponential hardware gains come from? Optimizing the software by 2x or 4x happens only once. Then there's the other side: if Moore's Law works too well, local models will be good enough for most tasks, and these companies won't be able to do the SaaS thing. It seems to me like models' capability scales logarithmically with size and wattage, making them the rare piece of software that can counteract Moore's Law. That doesn't seem like a way to make a trillion dollars.
- throwaway290 1y ago
  One improvement is from scraping and stealing better quality IP to train on. And they can just ride Moore's law until they profit then lobby governments to require licenses for fast GPUs because of national security.

Frieren 1y ago

> I think this is a minor speed bump and VC’s believe that cost of inference will decrease over time and this is a gold rush to grab market share while cost of inference declines. It could also be that you give too much credit to the market. People follow trends because in most cases that makes money. There is no other deeper though involved. Look at the financial crisis, totally irrational.

rglover 1y ago
This. Post-crypto, AI was the obvious next gambit for VC. Their money flows to hype, not product. The second that hype starts to fade and the money dries up, VCs will be running with their Harold Hill trunk full of cash toward the border. Just from the content they publish alone you can tell they're channeling their inner Barnum & Bailey in between Ayahuasca seizures.

xnx 1y ago

> inference costs will clearly drop and these companies will eventually be very profitable. Inference costs for old models will drop, but inference costs may stay the same if models continue to improve. No guarantee that any wrapper for inference will be able to hold on to customers when they stop selling $1.00 for $0.50.

deleted 1y ago
[deleted]

onlyrealcuzzo 1y ago

Isn't the consensus that the MOE architecture and other optimizations in the newest gen models (GPT-5, Gemini 3.0 to come, etc) will reduce inference costs by 50-75% already?

yomismoaqui 1y ago
Sounds interesting, do you have some links with more info about this? Thanks!
ACCount37 1y ago
Kind of. Frontier LLMs aren't going to get cheaper, but that's because the frontier keeps advancing. Price-performance though? The trend is clear: a given level of LLM capability keeps getting cheaper, and that trend is expected to hold. Improvements in architecture and training make LLMs more capability-dense, and advanced techniques make inference cheaper.
- onlyrealcuzzo 1y ago
  > Frontier LLMs aren't going to get cheaper One of the main selling points to MOE, is that the architecture is designed such that you can re-train experts independently, as well as add new experts, change the size of an experts parameters, etc, without retraining the entire model. If 80% of you usage comes from 20% of your experts, you can cut your future training costs SUBSTANTIALLY.
  - ACCount37 1y ago
    I don't think anyone has ever managed to pull off a success story with this kind of advanced expert manipulation. It's not entirely impossible, but I remain skeptical until see a proof that it, first, works. And, second, that it actually has an advantage over "we'll just train another base model from scratch, but 10% larger, with those +5% performance architecture tweaks, and a new modality blender, and more of that good highly curated data in the dataset, and fresher data overall, and it'll be glorious".

x0x0 1y ago

> inference costs will clearly drop They haven't though. On two fronts: 1, the soa models have been pretty constantly priced, and everyone wants the soa models. Likely the only way costs drop is the models get so good that people are like hey, I'm fine with a less useful answer (which is still good enough) and that seems, right now, like a bad bet. and 2 - we use a lot more tokens now. No more pasting Q&A into a site; now people hammer up chunks of their codebases and would love to push more. More context, more thinking, more everything.

ctoth 1y ago
You're describing increased spending while calling it increased cost. These aren't the same thing. A task that cost me $5 to accomplish with GPT-4 last year might cost $1 with Sonnet today, even though I'm now spending $100/month total on AI instead of $20 because I'm doing 100x more tasks. The cost per task dropped 80%. My spending went up 5x. Both statements are true. Here's an analogy you may understand: https://crespo.business/posts/cost-of-inference/ https://crespo.business/posts/cost-of-inference/
- KallDrexx 1y ago
  Fwiw that's not necessarily true, because if Sonnet ends up using reasoning, then you are using more tokens than GPT-4 would have used for the same task. Same with GPT-5 since it will decide (using an LLM) if it should use the thinking model for it (and you don't have as much control over it).
  - steveklabnik 1y ago
    This is addressed in the post.
    - x0x0 1y ago
      Right, but if I understand you, the counterargument is dumb since the context in which we are discussing is business viability (vcs investing in businesses where the unit economics require inference cost decreases), so actual dollars out rather than some imaginary cost per token is the metric that matters. Inference is getting so much cheaper that cursor and zed have had to raise prices.

Analemma_ 1y ago

My own usage and the usage of pretty much everyone I know says that as inference costs drop, usage goes up in lockstep, and I’m still nowhere near the ceiling of how many tokens I could use if they were free. I think if these companies are gambling their future on COGS going down, that’s a gamble they’re going to lose.

delaminator 1y ago
aka Jevons Paradox. we strive to make machines more efficient to reduce energy usage. That makes $ per unit of work lower. But instead of reducing spending, people find more uses for the machines.

username223 1y ago

Color me skeptical. We're running into the speed of light when it comes to transistor size, and the parallelism that made neural nets take off is running into power demands. Where do the exponential hardware gains come from? Optimizing the software by 2x or 4x happens only once. Then there's the other side: if Moore's Law works too well, local models will be good enough for most tasks, and these companies won't be able to do the SaaS thing. It seems to me like models' capability scales logarithmically with size and wattage, making them the rare piece of software that can counteract Moore's Law. That doesn't seem like a way to make a trillion dollars.

throwaway290 1y ago
One improvement is from scraping and stealing better quality IP to train on. And they can just ride Moore's law until they profit then lobby governments to require licenses for fast GPUs because of national security.