4 ms·

The model providers are not in the low margin part of the business. The unit economies of paid-per-token APIs are clearly favorable, and scale amazingly well as

by jsnell 1y ago

The model providers are not in the low margin part of the business. The unit economies of paid-per-token APIs are clearly favorable, and scale amazingly well as long as you can procure enough compute. I think it's the subscription-based models that are tricky to make work in the long term, since they suffer from adverse selection. Only the heaviest users will pay for a subscription, and those are the users that you either lose money on or make unhappy with strict usage limits. It's kind of the inverse of the gym membership model. Honestly, I think the subscriptions are mainly used as a demand moderation method for advanced features.

raincole 1y ago
> The model providers are not in the low margin part of the business. Many people believe that model providers are running at negative margin. (I don't know how true it is.)
- apothegm 1y ago
  They probably have been running at negative margin, or at the very least started that way. But between hardware and software developments, their cost structures are undoubtedly improving over time —- otherwise we wouldn’t be seeing pricing drop with each new generation of models. In fact, I would bet that their margins are improving in spite of the price drops.
- jsnell 1y ago
  Yes, many people believe that, but it doesn't seem to be an evidence-based belief. I've written about this in some detail[0][1] before. But since just linking to one's own writing is a bit gauche and doesn't make for a good discussion, I'll summarize :) 1. There is no point in providing paid APIs at negative margins, since there's no platform power in having a larger paid API share (paid access can't be used for training data, no lock-in effects, no network effects, no customer loyalty, no pricing power on the supply side since Nvidia doesn't give preferential treatment to large customers). Even selling access at break-even makes no sense, since that is just compute you're not using for training, or not selling to other companies desperate for compute. 2. There are 3rd-party providers selling only the compute, not models, who have even less reason to sell at a loss. Their prices are comparable to 1st-party providers. 3. Deepseek published their inference cost structure for R1. According to that data their paid API traffic is very lucrative (their GPU rental costs for inference are under 20% of their standard pricing, i.e. >80% operating margins; and the rental costs would cover power, cooling, depreciation of the capital investment). Insofar as frontier labs are unprofitable, I think it's primarily due to them giving out vast amounts of free access. [0] https://www.snellman.net/blog/archive/2025-06-02-llms-are-cheap/ https://www.snellman.net/blog/archive/2025-06-02-llms-are-ch... [1] https://news.ycombinator.com/item?id=44165521 https://news.ycombinator.com/item?id=44165521
  - what 1y ago
    There are more factors to cost than just the raw compute to provide inference. They can’t just fire everyone and continue to operate while paying just the compute cost. They also can’t stop training new models. The actual cost is much more than the compute for inference.
    - brookst 1y ago
      I heart you. Classic fixed / variable cost fallacy: if you look at the steel and plastic in a $200k Ferrari, it’s worth about $10k. They have 95% gross margins! Outrageous! (Nevermind the engine R&D cost, the pre-production molds that fail, the testing and marketing and product placement and…)
    - ummonk 1y ago
      That's all the more reason to run at a positive margin though - why shovel money into taking a loss on inference when you need to spend money on R&D?
    - jsnell 1y ago
      Yes, there are some additional operating costs, but they're really marginal compared to the cost of the compute. Your suggestion was personnel: Anthropic is reportedly on a run-rate of $3B with O(1k) employees, most of whom aren't directly doing ops. Likewise they also have to pay for non-compute infra, but it is a rounding error. Training is a fixed cost, not a variable cost. My initial comment was on the unit economics, so fixed costs don't matter. But including the full training costs doesn't actually change the math that much as far as I can tell for any of the popular models. E.g. the alleged leaked OpenAI financials for 2024 projected $4B spent on inference, $3B on training. And the inference workloads are currently growing insanely fast, meaning the training gets amortized over a larger volume of inference (e.g. Google showed a graph of their inference volume at Google I/O -- 50x growth in a year, now at 480T tokens / month[0]) [0] https://blog.google/technology/ai/io-2025-keynote/ https://blog.google/technology/ai/io-2025-keynote/
  - lmeyerov 1y ago
    I think you miss 2 big aspects: 1. High volume providers get efficiencies that low volume do not. It comes from both more workload giving more optimization opportunities, and staffing to do better engineering to begin with. The result is break even for lower volume firms is profitable for higher volume, and as high volume is magnitudes more scale, this quickly pays for many people. By being the high-volume API, this game can be played. If they choose not to bother, it is likely because strategic views on opportunity cost, not inability. That's not even the interesting analysis, which is what the real stock value is, or whatever corp structure scheme they're doing nowadays: 2. Growth for growths sake. Uber was exactly this kind of growth-at-all-costs play, going more into debt with every customer and fundraise. My understanding is they were able to tame costs and find side businesses (delivery, ...), with the threat becoming more about category shift of self-driving. By having the channel, they could be the one to monetize as that got figured out better. Whether tokens or something else becomes what is charged for at the profit layers (with breakeven tokens as cost of business), or subsidization ends and competitive pricing dominates, being the user interface to chat and the API interface to devs gives them channel. Historically, it is a lot of hubris to believe channel is worthless, and especially in an era of fast cloning.
mvdtnz 1y ago
What evidence do you have that there's decent margin on the APIs?
tonyhart7 1y ago
subscription model is just there to serve B2C side of business which in turn them into B2B side antrophic said themselves that enterprise is where the money at, but you cant just serve enterprise on the get go right this is where the B2C indirect influence comes
brookst 1y ago
Citation needed. Model providers spend a ton of money. It is unclear if they will ever have high margins. Today they are somewhere between zero and negative big numbers.

by jsnell 1y ago

raincole 1y ago

> The model providers are not in the low margin part of the business. Many people believe that model providers are running at negative margin. (I don't know how true it is.)

apothegm 1y ago
They probably have been running at negative margin, or at the very least started that way. But between hardware and software developments, their cost structures are undoubtedly improving over time —- otherwise we wouldn’t be seeing pricing drop with each new generation of models. In fact, I would bet that their margins are improving in spite of the price drops.
jsnell 1y ago
Yes, many people believe that, but it doesn't seem to be an evidence-based belief. I've written about this in some detail[0][1] before. But since just linking to one's own writing is a bit gauche and doesn't make for a good discussion, I'll summarize :) 1. There is no point in providing paid APIs at negative margins, since there's no platform power in having a larger paid API share (paid access can't be used for training data, no lock-in effects, no network effects, no customer loyalty, no pricing power on the supply side since Nvidia doesn't give preferential treatment to large customers). Even selling access at break-even makes no sense, since that is just compute you're not using for training, or not selling to other companies desperate for compute. 2. There are 3rd-party providers selling only the compute, not models, who have even less reason to sell at a loss. Their prices are comparable to 1st-party providers. 3. Deepseek published their inference cost structure for R1. According to that data their paid API traffic is very lucrative (their GPU rental costs for inference are under 20% of their standard pricing, i.e. >80% operating margins; and the rental costs would cover power, cooling, depreciation of the capital investment). Insofar as frontier labs are unprofitable, I think it's primarily due to them giving out vast amounts of free access. [0] https://www.snellman.net/blog/archive/2025-06-02-llms-are-cheap/ https://www.snellman.net/blog/archive/2025-06-02-llms-are-ch... [1] https://news.ycombinator.com/item?id=44165521 https://news.ycombinator.com/item?id=44165521
- what 1y ago
  There are more factors to cost than just the raw compute to provide inference. They can’t just fire everyone and continue to operate while paying just the compute cost. They also can’t stop training new models. The actual cost is much more than the compute for inference.
  - brookst 1y ago
    I heart you. Classic fixed / variable cost fallacy: if you look at the steel and plastic in a $200k Ferrari, it’s worth about $10k. They have 95% gross margins! Outrageous! (Nevermind the engine R&D cost, the pre-production molds that fail, the testing and marketing and product placement and…)
  - ummonk 1y ago
    That's all the more reason to run at a positive margin though - why shovel money into taking a loss on inference when you need to spend money on R&D?
  - jsnell 1y ago
    Yes, there are some additional operating costs, but they're really marginal compared to the cost of the compute. Your suggestion was personnel: Anthropic is reportedly on a run-rate of $3B with O(1k) employees, most of whom aren't directly doing ops. Likewise they also have to pay for non-compute infra, but it is a rounding error. Training is a fixed cost, not a variable cost. My initial comment was on the unit economics, so fixed costs don't matter. But including the full training costs doesn't actually change the math that much as far as I can tell for any of the popular models. E.g. the alleged leaked OpenAI financials for 2024 projected $4B spent on inference, $3B on training. And the inference workloads are currently growing insanely fast, meaning the training gets amortized over a larger volume of inference (e.g. Google showed a graph of their inference volume at Google I/O -- 50x growth in a year, now at 480T tokens / month[0]) [0] https://blog.google/technology/ai/io-2025-keynote/ https://blog.google/technology/ai/io-2025-keynote/
- lmeyerov 1y ago
  I think you miss 2 big aspects: 1. High volume providers get efficiencies that low volume do not. It comes from both more workload giving more optimization opportunities, and staffing to do better engineering to begin with. The result is break even for lower volume firms is profitable for higher volume, and as high volume is magnitudes more scale, this quickly pays for many people. By being the high-volume API, this game can be played. If they choose not to bother, it is likely because strategic views on opportunity cost, not inability. That's not even the interesting analysis, which is what the real stock value is, or whatever corp structure scheme they're doing nowadays: 2. Growth for growths sake. Uber was exactly this kind of growth-at-all-costs play, going more into debt with every customer and fundraise. My understanding is they were able to tame costs and find side businesses (delivery, ...), with the threat becoming more about category shift of self-driving. By having the channel, they could be the one to monetize as that got figured out better. Whether tokens or something else becomes what is charged for at the profit layers (with breakeven tokens as cost of business), or subsidization ends and competitive pricing dominates, being the user interface to chat and the API interface to devs gives them channel. Historically, it is a lot of hubris to believe channel is worthless, and especially in an era of fast cloning.

mvdtnz 1y ago

What evidence do you have that there's decent margin on the APIs?

tonyhart7 1y ago

subscription model is just there to serve B2C side of business which in turn them into B2B side antrophic said themselves that enterprise is where the money at, but you cant just serve enterprise on the get go right this is where the B2C indirect influence comes

brookst 1y ago

Citation needed. Model providers spend a ton of money. It is unclear if they will ever have high margins. Today they are somewhere between zero and negative big numbers.