4 ms·

> What worries me about this is that Anthropic and OpenAI seem to have backed themselves into a corner of high costs. Can they reasonably decrease their prices

> What worries me about this is that Anthropic and OpenAI seem to have backed themselves into a corner of high costs. Can they reasonably decrease their prices by 20-50x to compete with DeepSeek or Xiaomi’s Mimo? They have high prices, not high costs. They will obviously keep prices as high as they can for as long as they can, while keeping demand up. Once demand starts to fall, so will the prices. > Are these models cheap because they are open weight and having hundreds or people stress test running them on different hardware helped to lower the cost? Or is it that they are being provided as loss leaders to drive the prices down? Neither. They are cheap because they have neither technical edge nor brand power to keep the prices high, and so have to ask commodity prices for them. People somehow still don't get it, despite everyone who studies the economics of it telling them: Inference is dirt cheap. Training is expensive, inference is cheap, and getting cheaper.

Schiendelman 4mo ago
I get it! And I appreciate people like you pointing out the business side of LLMs. Also, these open weight models are significantly lower quality than the high end coding models, and for some reason a lot of people think they're exactly the same. Maybe engineers who only dabble in LLM usage aren't doing enough complex work to notice...?
manwithopinions 4mo ago
So why are they losing so much money? Money is made on the subset of inference that is charged at cost + margin via their APIs. API usage is so high because customers are still finding their feet, trying to understand how to measure the value they get from their spend, erring on the side of spend. Yes, in a world of unmeasured value and tokenmaxxing, inference is profitable on SOTA models because all capacity is being consumed at all times, driving down marginal costs, but what about a world in which capacity isn’t constrained? There are still huge fixed costs. Even the most optimistic leaks with the current high prices put the margin on API token inference at around 50%. How can SOTA models ever come close to competing on price? Price always matters. Offering the best model with the most brand recognition does not exempt OpenAI from the basic rules of business. Historically, software has been such a successful business because the margins are incredible, 95%+ in many cases, driven by direct measurable value to customers that dwarfs the cost. A 50% margin at a time when your customers are falling over themselves to spend as much money as they can is not a good sign, it is a very bad sign, it leaves no room to ever achieve traditional technology margins, and inevitably leads to very weak margins. Inference needs to become an order of magnitude cheaper than the value it delivers to ever have a chance of delivering on this wildly profitable vision. The cheap model providers have a much better chance of achieving that. Outside of coding, almost every business case for AI doesn’t need above human intelligence, it doesn’t even need human intelligence, or half a human intelligence, a business can extract a lot of value from a machine that has a fraction of a human’s intelligence. Most human work does not use our intelligence, it is rote, a monkey could do it, and that’s where AI will be used most. Who is going to pay $10 per million tokens when they could pay $0.10 to get the same outcomes?
- Onavo 4mo ago
  They are overly bloated organizations too. The human costs are tremendously high. Good luck finding ML engineers paid 7-8 figs USD in Asia. Same quality engineers, different market.
- Tuna-Fish 4mo ago
  > So why are they losing so much money? Mostly training. Claude didn't just get to be so good at coding by magic, it was suddenly so good because they did truly staggering amounts of RLHF and RLAIF on it. They are still doing that today, on any tasks they can figure out how to evaluate it on. This is capex for them. Their margins on inference are >90% today for tokens they sell (plans are hard to count, but still profitable). Based on what we know of it's size and architecture, running Opus is not more than 2x more expensive than running Deepseek v4 pro, for which tokens are available at under 10% of the cost of Opus. Again, the reason their margins are 50% is because they are spending so much on things that are not inference, not because inference is expensive. > The cheap model providers have a much better chance of achieving that. Anthropic can do it with a push of a button, once they calculate that it will provide them better profit than current pricing.
  - CSSer 4mo ago
    I think people miss this because these companies exist in a space that is new in tech, and that means lots of competition through PR and marketing. When that happens, it’s easy to feel like a company is telling you about everything they’ve been working on or are openly talking about what gives them their edge when in fact the opposite is often true.
  - manwithopinions 4mo ago
    > Their margins on inference are >90% today for tokens they sell (plans are hard to count, but still profitable). That doesn’t make any sense, it doesn’t add up. Have you seen how much money they’re raising and burning? We know that training does not cost tens of billions. Brockman said OpenAI expects to spend $50 billion on compute this year. OpenAI’s revenue run rate is less than $50 billion for this year! For 90% margins to be possible on inference, you are suggesting that less than $5 billion of that compute spend is inference and over $45 billion of that compute is training. Anthropic have been desperately trying to juggle capacity by shaping user behavior through peak time usage limits because they are struggling with capacity for inference. Plan based usage is widely acknowledged to be subsidized, you are probably the only person on earth suggesting that plans are profitable.
uejfiweun 4mo ago
What are you even talking about? Everyone knows that Anthropic is drastically subsidizing their plans. It's actually the exact opposite of what you're talking about. The costs are extremely high and the prices are actually what's being subsidized and cheap right now.
- Tuna-Fish 4mo ago
  This is an example of common knowledge that is wrong. People look at their cash burn, assume that they spend this to subsidize inference, and get bonkers answers. Inference is not their largest expense. Inference is cheap. Anthropic is only drastically subsidizing their plans if you count their training expenses as part of their costs.
  - therobots927 4mo ago
    Are you an anthropic insider or something? Because if you are you should delete this comment. If you aren’t then you don’t know what the hell you’re talking about.
    - xquce 3mo ago
      Surely the same can be said for the people saying the opposite?
      - therobots927 3mo ago
        I didn’t make a claim. The parent explicitly said it was a misconception that inference is not profitable. No one knows if it’s profitable or not so we’re left to speculate.
        Bnjoroge 3mo ago
        You can make a fairly decent assumption by calculating the margin on serving glm 5.2, and adding say 30% extra costs and it still leaves a healthy margin
amanaplanacanal 3mo ago
You can't just define training as "not a cost". Without training they have nothing to sell.
danny_codes 3mo ago
So they reduce their prices 50x and their revenue drops to $1B a year? And they're supposed to be worth $1T each? If there is an open model offering at the cost of inference + whatever margin you need to stay alive, then as inference falls Anthropic/OpenAI revenue also falls. What's the end-game? It sounds like in this business, if you are right, your revenues drop every year.

Schiendelman 4mo ago

I get it! And I appreciate people like you pointing out the business side of LLMs. Also, these open weight models are significantly lower quality than the high end coding models, and for some reason a lot of people think they're exactly the same. Maybe engineers who only dabble in LLM usage aren't doing enough complex work to notice...?

manwithopinions 4mo ago

So why are they losing so much money? Money is made on the subset of inference that is charged at cost + margin via their APIs. API usage is so high because customers are still finding their feet, trying to understand how to measure the value they get from their spend, erring on the side of spend. Yes, in a world of unmeasured value and tokenmaxxing, inference is profitable on SOTA models because all capacity is being consumed at all times, driving down marginal costs, but what about a world in which capacity isn’t constrained? There are still huge fixed costs. Even the most optimistic leaks with the current high prices put the margin on API token inference at around 50%. How can SOTA models ever come close to competing on price? Price always matters. Offering the best model with the most brand recognition does not exempt OpenAI from the basic rules of business. Historically, software has been such a successful business because the margins are incredible, 95%+ in many cases, driven by direct measurable value to customers that dwarfs the cost. A 50% margin at a time when your customers are falling over themselves to spend as much money as they can is not a good sign, it is a very bad sign, it leaves no room to ever achieve traditional technology margins, and inevitably leads to very weak margins. Inference needs to become an order of magnitude cheaper than the value it delivers to ever have a chance of delivering on this wildly profitable vision. The cheap model providers have a much better chance of achieving that. Outside of coding, almost every business case for AI doesn’t need above human intelligence, it doesn’t even need human intelligence, or half a human intelligence, a business can extract a lot of value from a machine that has a fraction of a human’s intelligence. Most human work does not use our intelligence, it is rote, a monkey could do it, and that’s where AI will be used most. Who is going to pay $10 per million tokens when they could pay $0.10 to get the same outcomes?

Onavo 4mo ago
They are overly bloated organizations too. The human costs are tremendously high. Good luck finding ML engineers paid 7-8 figs USD in Asia. Same quality engineers, different market.
Tuna-Fish 4mo ago
> So why are they losing so much money? Mostly training. Claude didn't just get to be so good at coding by magic, it was suddenly so good because they did truly staggering amounts of RLHF and RLAIF on it. They are still doing that today, on any tasks they can figure out how to evaluate it on. This is capex for them. Their margins on inference are >90% today for tokens they sell (plans are hard to count, but still profitable). Based on what we know of it's size and architecture, running Opus is not more than 2x more expensive than running Deepseek v4 pro, for which tokens are available at under 10% of the cost of Opus. Again, the reason their margins are 50% is because they are spending so much on things that are not inference, not because inference is expensive. > The cheap model providers have a much better chance of achieving that. Anthropic can do it with a push of a button, once they calculate that it will provide them better profit than current pricing.
- CSSer 4mo ago
  I think people miss this because these companies exist in a space that is new in tech, and that means lots of competition through PR and marketing. When that happens, it’s easy to feel like a company is telling you about everything they’ve been working on or are openly talking about what gives them their edge when in fact the opposite is often true.
- manwithopinions 4mo ago
  > Their margins on inference are >90% today for tokens they sell (plans are hard to count, but still profitable). That doesn’t make any sense, it doesn’t add up. Have you seen how much money they’re raising and burning? We know that training does not cost tens of billions. Brockman said OpenAI expects to spend $50 billion on compute this year. OpenAI’s revenue run rate is less than $50 billion for this year! For 90% margins to be possible on inference, you are suggesting that less than $5 billion of that compute spend is inference and over $45 billion of that compute is training. Anthropic have been desperately trying to juggle capacity by shaping user behavior through peak time usage limits because they are struggling with capacity for inference. Plan based usage is widely acknowledged to be subsidized, you are probably the only person on earth suggesting that plans are profitable.

uejfiweun 4mo ago

What are you even talking about? Everyone knows that Anthropic is drastically subsidizing their plans. It's actually the exact opposite of what you're talking about. The costs are extremely high and the prices are actually what's being subsidized and cheap right now.

Tuna-Fish 4mo ago
This is an example of common knowledge that is wrong. People look at their cash burn, assume that they spend this to subsidize inference, and get bonkers answers. Inference is not their largest expense. Inference is cheap. Anthropic is only drastically subsidizing their plans if you count their training expenses as part of their costs.
- therobots927 4mo ago
  Are you an anthropic insider or something? Because if you are you should delete this comment. If you aren’t then you don’t know what the hell you’re talking about.
  - xquce 3mo ago
    Surely the same can be said for the people saying the opposite?
    - therobots927 3mo ago
      I didn’t make a claim. The parent explicitly said it was a misconception that inference is not profitable. No one knows if it’s profitable or not so we’re left to speculate.
      - Bnjoroge 3mo ago
        You can make a fairly decent assumption by calculating the margin on serving glm 5.2, and adding say 30% extra costs and it still leaves a healthy margin

amanaplanacanal 3mo ago

You can't just define training as "not a cost". Without training they have nothing to sell.

danny_codes 3mo ago

So they reduce their prices 50x and their revenue drops to $1B a year? And they're supposed to be worth $1T each? If there is an open model offering at the cost of inference + whatever margin you need to stay alive, then as inference falls Anthropic/OpenAI revenue also falls. What's the end-game? It sounds like in this business, if you are right, your revenues drop every year.