6 ms·

Here are a few thoughts: - The publicly available information about how inference costs compare to training costs is conflicted. EEs involved in datacenters ta

Here are a few thoughts: - The publicly available information about how inference costs compare to training costs is conflicted. EEs involved in datacenters talk about power usage spikes during training runs as if they were a major factor in the designs, but academic papers discussing cost-optimal scaling confidently treat inference-time compute as a major factor. - On the side of the balance indicating that training is more compute-intensive after amortization than inference is that Chinese providers, constrained primarily by access to compute, have nearly unlimited token availability at a lower price than US providers (inference), but poorer model capabilities (training). That would make sense only if US providers are inflating inference costs by 20-30x due to amortized training costs that overseas providers were not able to take on (there are other factors too). - If training >> inference, they're in a prisoner's dilemma that far exceeds the ordinary zero-marginals model of competition between firms (due to its huge discrete stepwise nature). On the other hand, if inference>>training, the high-level analysis popularized by certain thought leaders, that it's like a utility, would be true. You'd tend to count this as a vote for inference>>training, but the CEOs saying it at least have a huge incentive to agree because the alternative, the prisoner's dilemma, would stop investment very fast. - The only voice in the story that I just told you to have anything to do with fact (as opposed to high-level analysis and ivory tower armchair management of a secretive business) were the rumors from facilities engineers. That shows you the state of our understanding... - If we don't even know the ratio between amortized capital expenses and operational costs, outside investor analysis is impossible. It doesn't matter how finely they divide the accounting buckets for office ferns and indoor ferns if the single biggest part of their business is obscured for trade secret reasons.

materielle 5mo ago
I'm about to leave a shallow comment, but I am a bit skeptical of the supposed drop in inference costs. If AI labs saw a lot of potential there, they'd surely be bragging about it non-stop? So the fact that publicly available information is conflicted is probably a sign that at the very least, the numbers aren't amazing. Yes I know there's no evidence and this is lazy reasoning. But there's probably a bit of truth to this line of thought.
- whatshisface 5mo ago
  Inference has traditionally been far less expensive than training. One public example is the fact that hobbyists can run StableDiffusion ($600k training costs[1]) on their personal computers. Speaking to your point, inference being dramatically less costly than training would not be seen as a delta from the norm. The model of providing inference for anything near the operational costs (like a utility would), would the delta from the norm if it were true. [1] https://x.com/emostaque/status/1563870674111832066 https://x.com/emostaque/status/1563870674111832066
  - thesz 5mo ago
    The difference between training and inference is 1) one have to keep intermediate results for backward pass in training and 2) computation for training double because of the backward pass. Training is also done over batches, which increase memory requirements by several orders of magnitude. This is why training needs costly compute. One of the ways out of this unfortunate situation is to use something like Stochastic Average Gradient Descent [1]. Examples there are mostly concerned with regularized logistic regression, which makes problem more or less convex. Neural networks are inherently non-convex. Still, maybe some ideas from there can be utilized in the context of neural networks, like use of estimated Lipshitz constant to derive curvature and appropriate learning step. [1] https://www.cs.ubc.ca/~schmidtm/Courses/540-W19/L12.pdf
    - janalsncm 5mo ago
      So one way to think about it is roughly, Training is inference + backwards pass (~2x inference cost) + activations (vram overhead) + optimizer (vram overhead) + gradients (vram overhead).
      - thesz 5mo ago
        Multiply "inference + backwards pass (~2x inference cost) + activations (vram overhead)" by batch size (thousands) to get to the actual RAM and compute cost. Optimizer like ADAM adds only two or three model-sized overhead. And last, but not least, you need only one hidden layer kept in RAM for inference, but you need all of them (61 for Deepseek models) kept in RAM for computing gradient for one sample.
vlovich123 5mo ago
Small alternative potential future changes that alter this analysis: * At some point model capability reaches diminishing returns. Then inference >> training in the future but training >> inference now. It’s not a prisoner’s dilemma but a land grab to solidify market position and be one of the 2-3 firms left standing as dominant in the space. The model companies aren’t super sticky yet but they’re working on it. * even if training remains >> inference, it’s possible to have multiple price points like they do today. If you need the most capable model you’ll be paying exponentially more per token to supplement the training cost even though the serving cost is marginal because most people will be satisfied with cheaper / less capable models for most tasks. I buy that inference is a dropping line item while training is a growing one. There’s all sorts of things on the horizon that’ll be order of magnitudes improvements, from startups burning models into ASICs to get order of magnitudes more performance to alternate architectures like diffusion transformers that have orders of magnitude structural optimizations. It’s inevitable that it’ll come down even further from where we are. It’s possible model training also will go down but I’ve not seen any compelling research suggesting major “easy” reductions here.
- janalsncm 5mo ago
  The issue is that most tasks do not require frontier-level intelligence, but companies like OAI can really only profit off of the frontier. Capabilities from a year or two ago are so outdated that even OpenAI gives it away for free and there are many other models biting at their heels. In other words they are spending huge amounts of money to cash in on a depreciating asset. So one possible future is that frontier-level training becomes so expensive and the use cases so sparse that it simply isn’t viable to keep going bigger.
  - extraextra 5mo ago
    Once the land grab is over, the market will consolidate and the winners will absorb the losers. Then the few winners will be the only ones with real capital to train frontier models and will have true pricing power. Similar to how social media companies or the gig-economy benefits from network effects, AI companies will benefit from having the lion's share of paying customers (that also constantly feed in more data to train the models on).
somewhereoutth 5mo ago
Yes the huge discrete stepwise training spend is critical. Maybe investors will realise that "the only winning move is not to play". And so we are left with (as was) frontier models getting more and more out of date as whoever their post bankruptcy custodians are tries to eek pennies on the dollar for inference on their decaying property. Perhaps along with local and/or highly specialized models still feeding on the after-glow of the huge amount of training that was (and is no longer) done. The next AI winter is going to be deep, savage, and long.
- galaxyLogic 5mo ago
  > frontier models getting more and more out of date Why are they getting out of date? Is it because we have new content from the internet that the older models did not have? Or are we simply trying to increase the size of the training data? In other words not more up-todate in terms of time the content was created vs. wanting to use bigger training-input-sets?
  - somewhereoutth 5mo ago
    lack of new content from the internet will make them go out of date. Not just facts and figures, but (for example) new programming languages/techniques.
    - galaxyLogic 5mo ago
      I see makes sense. Then it it kinda says that quality of the model is the topicality of its inputs I assume.
- extraextra 5mo ago
  Bankruptcies? The winners will gobble up the losers and the few remaining players will have pricing power. Don't be naive thinking that OpenAI or Anthropic can possibly go bankrupt. There will always be someone happy to buy them up for a nice price. Yes, the market will have to go through a consolidation phase though.
stevenally 5mo ago
> If we don't even know the ratio between amortized capital expenses and operational costs, outside investor analysis is impossible. And yet we surely need this data for the IPO? Or are they relying on rule changes on the indexes to force ETFs to buy shares?
- extraextra 5mo ago
  The IPOs are months away, potentially 6 months or more. We're in a volatile macro environment. AI companies have all the incentives to not create higher expectations regarding their financial situation a long time before the IPO. Obviously at IPO they will have to disclose their full financial situation. The market is super hyped anyway for their IPOs. If they raise investors expectations now and things change until the IPO, investors will be disappointed. It's a lose-lose proposition. The smart play for any company is to keep their cards close to their chest until close to the IPO time.
twobitshifter 5mo ago
We have GPU costs, power costs, and how many token/s models can generate on those GPUs. It’s possible to figure out the marginal cost based on this. The current estimate is about $0.40 per million tokens for gpt4 equivalent model. Sonnet 4 is $15 per million tokens, so they are charging high margins on inference. The issue is how large of a margin is needed to recover their costs before the GPUs age out, and how high of a margin can be charged before it’s not economically viable. https://www.gpunex.com/blog/ai-inference-economics-2026/ https://www.gpunex.com/blog/ai-inference-economics-2026/
- rudedogg 5mo ago
  That seems way off to me. I skimmed the article, but couldn’t spot any details on their estimates. They mention 70b+ params as being large in several places. But we’ve had several 100b+ param models that trail Sonnet.
IX-103 5mo ago
I don't see how it would be possible for inference costs to dominate training costs, even after amortization. Training involves multiple passes over the entire training dataset, ideally in large batches where you can perform inference on as many samples as possible simultaneously and then perform backpropagation to adjust the model weights (which is about as expensive as inference). Let's consider the size of the dataset we're dealing with here. The dataset likely consists of practically every piece of digitized text they can get their hands on (including that extracted from audio and video). We know Google has digitized a large portion of the books in existence as part of their "search book contents" feature and we have no reason to believe they're not using it alongside their cache of 90+% of the internet to train their models. We're talking about 100s of millions of books each with an average of 100,000s of tokens. The internet has 10s to 100s of billions of pages on it with who knows how many tokens on average. This is a huge dataset that we've got to go through hundreds of times. Second, let's consider the effect of batching and how it sets requirements for our hardware. We know that larger batch sizes converge faster, are more stable, and produce better models. So if you want a good model you need large batch sizes. This means that you need machines several orders of magnitude more powerful than you use for inference. From what I heard Google uses clusters of 100s of the their TPUs all located in a single rack for training. These clusters are organized in a customized computing architecture to maximize memory locality between cores (really critical for efficient back-propagation). Further, you can't use reduced precision weights for training like you can for inference, so there are no shortcuts. Finally, the initial training stage is followed by reinforcement learning stages - this is key development in how AI models have improved in the past year. This may mean going through a curated set of traces (either synthetic or captured from users) and adjusting the weights based on experienced outcome. Overall there's so many orders of magnitude more work and more hardware requirements for training that I find it improbable that inference dominates. The number of "inference" steps in training is freaking ridiculous and includes such factors as the "number of words ever written".
- atq2119 5mo ago
  It's been a while since I saw a detailed paper on a high end training run, but extrapolating from what I remember, it seems those training runs are in the 10s of trillions of tokens. This already accounts for potentially sampling tokens multiple times during the training run. That seems like a large number, until you realize that OpenAI claims to have almost a billion weekly users. And OpenRouter shows many models at over a trillion tokens per week. So in pure token terms, I'd say it is in fact extremely plausible that inference dominates, at least for the popular models.
zozbot234 5mo ago
Why would power spikes from training runs imply training>>inference? The cost of a training run scales with energy, whereas power is energy per unit time. All that tells you is that they're speeding up their training run so it will take less time overall (probably chasing some first-mover advantage, where they're out with a given model before their closest competitors), whereas they obviously can't do that for inference (which is a steady flow of requests over time).
extraextra 5mo ago
Great points. - At the end of the day those are still private companies (albeit huge ones), so we can only speculate about the state of their private financial situation. Once they will decide it's the right time to IPO, they'll publish all their financials and we'll start to have a clearer picture. - Later, each company will slightly specialize and have a different go-to-market strategy, which will allow us to understand on a deeper level what works in the market and what doesn't (think about how Facebook, Instagram and TikTok are all huge universal social media platforms, but, each with a different target audience and different user base). - Finally, the market will go through a consolidation phase in which winners will gobble up the losers and then the incumbents will have a real moat (against new-comers) and real pricing power on their user base.

materielle 5mo ago

I'm about to leave a shallow comment, but I am a bit skeptical of the supposed drop in inference costs. If AI labs saw a lot of potential there, they'd surely be bragging about it non-stop? So the fact that publicly available information is conflicted is probably a sign that at the very least, the numbers aren't amazing. Yes I know there's no evidence and this is lazy reasoning. But there's probably a bit of truth to this line of thought.

whatshisface 5mo ago
Inference has traditionally been far less expensive than training. One public example is the fact that hobbyists can run StableDiffusion ($600k training costs[1]) on their personal computers. Speaking to your point, inference being dramatically less costly than training would not be seen as a delta from the norm. The model of providing inference for anything near the operational costs (like a utility would), would the delta from the norm if it were true. [1] https://x.com/emostaque/status/1563870674111832066 https://x.com/emostaque/status/1563870674111832066
- thesz 5mo ago
  The difference between training and inference is 1) one have to keep intermediate results for backward pass in training and 2) computation for training double because of the backward pass. Training is also done over batches, which increase memory requirements by several orders of magnitude. This is why training needs costly compute. One of the ways out of this unfortunate situation is to use something like Stochastic Average Gradient Descent [1]. Examples there are mostly concerned with regularized logistic regression, which makes problem more or less convex. Neural networks are inherently non-convex. Still, maybe some ideas from there can be utilized in the context of neural networks, like use of estimated Lipshitz constant to derive curvature and appropriate learning step. [1] https://www.cs.ubc.ca/~schmidtm/Courses/540-W19/L12.pdf
  - janalsncm 5mo ago
    So one way to think about it is roughly, Training is inference + backwards pass (~2x inference cost) + activations (vram overhead) + optimizer (vram overhead) + gradients (vram overhead).
    - thesz 5mo ago
      Multiply "inference + backwards pass (~2x inference cost) + activations (vram overhead)" by batch size (thousands) to get to the actual RAM and compute cost. Optimizer like ADAM adds only two or three model-sized overhead. And last, but not least, you need only one hidden layer kept in RAM for inference, but you need all of them (61 for Deepseek models) kept in RAM for computing gradient for one sample.

vlovich123 5mo ago

Small alternative potential future changes that alter this analysis: * At some point model capability reaches diminishing returns. Then inference >> training in the future but training >> inference now. It’s not a prisoner’s dilemma but a land grab to solidify market position and be one of the 2-3 firms left standing as dominant in the space. The model companies aren’t super sticky yet but they’re working on it. * even if training remains >> inference, it’s possible to have multiple price points like they do today. If you need the most capable model you’ll be paying exponentially more per token to supplement the training cost even though the serving cost is marginal because most people will be satisfied with cheaper / less capable models for most tasks. I buy that inference is a dropping line item while training is a growing one. There’s all sorts of things on the horizon that’ll be order of magnitudes improvements, from startups burning models into ASICs to get order of magnitudes more performance to alternate architectures like diffusion transformers that have orders of magnitude structural optimizations. It’s inevitable that it’ll come down even further from where we are. It’s possible model training also will go down but I’ve not seen any compelling research suggesting major “easy” reductions here.

janalsncm 5mo ago
The issue is that most tasks do not require frontier-level intelligence, but companies like OAI can really only profit off of the frontier. Capabilities from a year or two ago are so outdated that even OpenAI gives it away for free and there are many other models biting at their heels. In other words they are spending huge amounts of money to cash in on a depreciating asset. So one possible future is that frontier-level training becomes so expensive and the use cases so sparse that it simply isn’t viable to keep going bigger.
- extraextra 5mo ago
  Once the land grab is over, the market will consolidate and the winners will absorb the losers. Then the few winners will be the only ones with real capital to train frontier models and will have true pricing power. Similar to how social media companies or the gig-economy benefits from network effects, AI companies will benefit from having the lion's share of paying customers (that also constantly feed in more data to train the models on).

somewhereoutth 5mo ago

Yes the huge discrete stepwise training spend is critical. Maybe investors will realise that "the only winning move is not to play". And so we are left with (as was) frontier models getting more and more out of date as whoever their post bankruptcy custodians are tries to eek pennies on the dollar for inference on their decaying property. Perhaps along with local and/or highly specialized models still feeding on the after-glow of the huge amount of training that was (and is no longer) done. The next AI winter is going to be deep, savage, and long.

galaxyLogic 5mo ago
> frontier models getting more and more out of date Why are they getting out of date? Is it because we have new content from the internet that the older models did not have? Or are we simply trying to increase the size of the training data? In other words not more up-todate in terms of time the content was created vs. wanting to use bigger training-input-sets?
- somewhereoutth 5mo ago
  lack of new content from the internet will make them go out of date. Not just facts and figures, but (for example) new programming languages/techniques.
  - galaxyLogic 5mo ago
    I see makes sense. Then it it kinda says that quality of the model is the topicality of its inputs I assume.
extraextra 5mo ago
Bankruptcies? The winners will gobble up the losers and the few remaining players will have pricing power. Don't be naive thinking that OpenAI or Anthropic can possibly go bankrupt. There will always be someone happy to buy them up for a nice price. Yes, the market will have to go through a consolidation phase though.

stevenally 5mo ago

> If we don't even know the ratio between amortized capital expenses and operational costs, outside investor analysis is impossible. And yet we surely need this data for the IPO? Or are they relying on rule changes on the indexes to force ETFs to buy shares?

extraextra 5mo ago
The IPOs are months away, potentially 6 months or more. We're in a volatile macro environment. AI companies have all the incentives to not create higher expectations regarding their financial situation a long time before the IPO. Obviously at IPO they will have to disclose their full financial situation. The market is super hyped anyway for their IPOs. If they raise investors expectations now and things change until the IPO, investors will be disappointed. It's a lose-lose proposition. The smart play for any company is to keep their cards close to their chest until close to the IPO time.

twobitshifter 5mo ago

We have GPU costs, power costs, and how many token/s models can generate on those GPUs. It’s possible to figure out the marginal cost based on this. The current estimate is about $0.40 per million tokens for gpt4 equivalent model. Sonnet 4 is $15 per million tokens, so they are charging high margins on inference. The issue is how large of a margin is needed to recover their costs before the GPUs age out, and how high of a margin can be charged before it’s not economically viable. https://www.gpunex.com/blog/ai-inference-economics-2026/ https://www.gpunex.com/blog/ai-inference-economics-2026/

rudedogg 5mo ago
That seems way off to me. I skimmed the article, but couldn’t spot any details on their estimates. They mention 70b+ params as being large in several places. But we’ve had several 100b+ param models that trail Sonnet.

IX-103 5mo ago

I don't see how it would be possible for inference costs to dominate training costs, even after amortization. Training involves multiple passes over the entire training dataset, ideally in large batches where you can perform inference on as many samples as possible simultaneously and then perform backpropagation to adjust the model weights (which is about as expensive as inference). Let's consider the size of the dataset we're dealing with here. The dataset likely consists of practically every piece of digitized text they can get their hands on (including that extracted from audio and video). We know Google has digitized a large portion of the books in existence as part of their "search book contents" feature and we have no reason to believe they're not using it alongside their cache of 90+% of the internet to train their models. We're talking about 100s of millions of books each with an average of 100,000s of tokens. The internet has 10s to 100s of billions of pages on it with who knows how many tokens on average. This is a huge dataset that we've got to go through hundreds of times. Second, let's consider the effect of batching and how it sets requirements for our hardware. We know that larger batch sizes converge faster, are more stable, and produce better models. So if you want a good model you need large batch sizes. This means that you need machines several orders of magnitude more powerful than you use for inference. From what I heard Google uses clusters of 100s of the their TPUs all located in a single rack for training. These clusters are organized in a customized computing architecture to maximize memory locality between cores (really critical for efficient back-propagation). Further, you can't use reduced precision weights for training like you can for inference, so there are no shortcuts. Finally, the initial training stage is followed by reinforcement learning stages - this is key development in how AI models have improved in the past year. This may mean going through a curated set of traces (either synthetic or captured from users) and adjusting the weights based on experienced outcome. Overall there's so many orders of magnitude more work and more hardware requirements for training that I find it improbable that inference dominates. The number of "inference" steps in training is freaking ridiculous and includes such factors as the "number of words ever written".

atq2119 5mo ago
It's been a while since I saw a detailed paper on a high end training run, but extrapolating from what I remember, it seems those training runs are in the 10s of trillions of tokens. This already accounts for potentially sampling tokens multiple times during the training run. That seems like a large number, until you realize that OpenAI claims to have almost a billion weekly users. And OpenRouter shows many models at over a trillion tokens per week. So in pure token terms, I'd say it is in fact extremely plausible that inference dominates, at least for the popular models.

zozbot234 5mo ago

Why would power spikes from training runs imply training>>inference? The cost of a training run scales with energy, whereas power is energy per unit time. All that tells you is that they're speeding up their training run so it will take less time overall (probably chasing some first-mover advantage, where they're out with a given model before their closest competitors), whereas they obviously can't do that for inference (which is a steady flow of requests over time).

extraextra 5mo ago

Great points. - At the end of the day those are still private companies (albeit huge ones), so we can only speculate about the state of their private financial situation. Once they will decide it's the right time to IPO, they'll publish all their financials and we'll start to have a clearer picture. - Later, each company will slightly specialize and have a different go-to-market strategy, which will allow us to understand on a deeper level what works in the market and what doesn't (think about how Facebook, Instagram and TikTok are all huge universal social media platforms, but, each with a different target audience and different user base). - Finally, the market will go through a consolidation phase in which winners will gobble up the losers and then the incumbents will have a real moat (against new-comers) and real pricing power on their user base.