3 ms·

I don't think anyone has a firm grasp on actual inference costs -- including the research and training that has gone into those models. We've got near-frontier

I don't think anyone has a firm grasp on actual inference costs -- including the research and training that has gone into those models. We've got near-frontier capabilities from open source models from China at pennies on the dollar compared to US big tech rollouts. OpenAI and Anthropic are heavily subsidizing their inference -- no wait, they are charging the most they can get away with before going public. Where is the truth?

MichaelMedbed 4mo ago
[flagged]
- polski-g 4mo ago
  Not relevant to the post.
- kllrnohj 4mo ago
  regardless of whether that's true or not, US companies doing hosted inference of the models coming out of China are also significantly cheaper than those from OpenAI or Anthropic
andrewmutz 4mo ago
Both can be true. They can be charging what the market will bear, and still be charging less than their costs of running it.
- wyre 4mo ago
  There is no way I'm believing DeepSeek can charge less than $1 USD for their pro model while Opus costs over 25x more, yet their price is less than the cost of running it?
  - kube-system 4mo ago
    It would seem strange, if they were operating in the same economy, but they don't. DeepSeek operates in an economy with a high degree of central planning. China subsidizes strategic industries, and they have heavily done so with AI. And DeepSeek specifically has said they have no commercialization plans. For example: https://www.boc.cn/aboutboc/bi1/202501/t20250123_25254674.html https://www.boc.cn/aboutboc/bi1/202501/t20250123_25254674.ht...
    - deleted 4mo ago
      [deleted]
    - wyrdcurt 4mo ago
      DeepSeek is not the only provider of inference for their models. Chinese subsidies likely do explain DeepSeek's ability to provide inference cheaper than other providers, but even a US provider like DeepInfra can serve DeepSeek 4 Pro at $1.30/M in and $2.60/M out. Unless American labs are doing something wildly inefficient, it feels safe to assume Anthropic has some profit margin on inference at API prices.
      - kube-system 4mo ago
        They may, neglecting overhead R&D. But also, some suspect that US models are significantly heavier than DeepSeek in resource consumption by multiple measures It’s generally established that Anthropic/OpenAI are going for all out performance with big VC dollars at the expense of efficiency and China has geopolitically limited compute and an inventive to compete on value per dollar.
  - re-thc 4mo ago
    > There is no way I'm believing DeepSeek can charge less Why not? Hetzner charges WAY less than AWS too. Can you not believe that?
    - orangecat 4mo ago
      That's the point. Hetzner is presumably covering their costs, so it's a safe bet that AWS is profitable.
pimeys 4mo ago
We pay by token at work. I just finished one session with Opus that was 4000 dollars. In about three days. Now that 200USD subscription starts to feel cheap...
- zozbot234 4mo ago
  That would be about ~300 tok/s over 72 hours at Claude Fable output token prices? I'm not sure that this passes a sanity test.
  - unholiness 4mo ago
    Subagents are a helluva drug.
- rubyn00bie 4mo ago
  Just outta curiosity, as I’ve never gotten a spend anywhere near that, what variant were you using? Like max context window and fast mode? Or was it just chugging along non stop for three days?
  - pimeys 4mo ago
    Fast mode max content window. The task was: replace all 1600+ queries from one database to another and make the whole integration test pass. We did multiple passes, with different concerns when changing from database to another. My OpenCode session right now says $4,365.02. I haven't gotten close to this either before, but now we wanted to move fast because this branch gets conflicts all the time and we want to get over with the migration asap.
    - rglullis 4mo ago
      It's a bit of a left field question, but I am curious: Let's say that if the company wasn't paying the whole bill but only subsidizing it - e.g, if it paid 90% of the $4000. What would you do?
      - 4mo ago
dontlikeyoueith 4mo ago
> OpenAI and Anthropic are heavily subsidizing their inference -- no wait, they are charging the most they can get away with before going public. Where is the truth? Both. They are charging the most they can get away with and that amount is still heavily subsidized by VC capital.
schaefer 4mo ago
> I don't think anyone has a firm grasp on actual inference costs. There are huge numbers of users (myself included) that do have an exact idea of what inference costs are - on open models. We can buy tokens from 3rd parties that have no motivation to subsidize our use. That's to say, there's a fair marketplace[1] and we're hanging out there. If you want to say "I don't think anyone has a firm grasp on actual inference costs on these proprietary/closed models", then I could agree with that. [1]: https://openrouter.ai/rankings#leaderboard https://openrouter.ai/rankings#leaderboard
logicchains 4mo ago
We have a firm grasp on actual inference costs from the various open weights model providers on OpenRouter. They don't have the money to subsidize inference and it's quite a competitive market, so the prices are representative of the costs.
InsideOutSanta 4mo ago
> I don't think anyone has a firm grasp on actual inference costs -- including the research and training that has gone into those models We know roughly how much these companies spend and what their revenues are. Based on that, they'd have to more than double revenue (without spending more money) just to stay even, and that's not good enough given how deep in the hole they are. > OpenAI and Anthropic are heavily subsidizing their inference -- no wait, they are charging the most they can get away with before going public. Where is the truth? Both are true. I mean, I'd be willing to spend a bit more than I do now, but not more than double, and neither are most companies. The company I work for is currently investigating how to reduce LLM spend, not looking to spend more.

MichaelMedbed 4mo ago

[flagged]

polski-g 4mo ago
Not relevant to the post.
kllrnohj 4mo ago
regardless of whether that's true or not, US companies doing hosted inference of the models coming out of China are also significantly cheaper than those from OpenAI or Anthropic

andrewmutz 4mo ago

Both can be true. They can be charging what the market will bear, and still be charging less than their costs of running it.

wyre 4mo ago
There is no way I'm believing DeepSeek can charge less than $1 USD for their pro model while Opus costs over 25x more, yet their price is less than the cost of running it?
- kube-system 4mo ago
  It would seem strange, if they were operating in the same economy, but they don't. DeepSeek operates in an economy with a high degree of central planning. China subsidizes strategic industries, and they have heavily done so with AI. And DeepSeek specifically has said they have no commercialization plans. For example: https://www.boc.cn/aboutboc/bi1/202501/t20250123_25254674.html https://www.boc.cn/aboutboc/bi1/202501/t20250123_25254674.ht...
  - deleted 4mo ago
    [deleted]
  - wyrdcurt 4mo ago
    DeepSeek is not the only provider of inference for their models. Chinese subsidies likely do explain DeepSeek's ability to provide inference cheaper than other providers, but even a US provider like DeepInfra can serve DeepSeek 4 Pro at $1.30/M in and $2.60/M out. Unless American labs are doing something wildly inefficient, it feels safe to assume Anthropic has some profit margin on inference at API prices.
    - kube-system 4mo ago
      They may, neglecting overhead R&D. But also, some suspect that US models are significantly heavier than DeepSeek in resource consumption by multiple measures It’s generally established that Anthropic/OpenAI are going for all out performance with big VC dollars at the expense of efficiency and China has geopolitically limited compute and an inventive to compete on value per dollar.
- re-thc 4mo ago
  > There is no way I'm believing DeepSeek can charge less Why not? Hetzner charges WAY less than AWS too. Can you not believe that?
  - orangecat 4mo ago
    That's the point. Hetzner is presumably covering their costs, so it's a safe bet that AWS is profitable.

pimeys 4mo ago

We pay by token at work. I just finished one session with Opus that was 4000 dollars. In about three days. Now that 200USD subscription starts to feel cheap...

zozbot234 4mo ago
That would be about ~300 tok/s over 72 hours at Claude Fable output token prices? I'm not sure that this passes a sanity test.
- unholiness 4mo ago
  Subagents are a helluva drug.
rubyn00bie 4mo ago
Just outta curiosity, as I’ve never gotten a spend anywhere near that, what variant were you using? Like max context window and fast mode? Or was it just chugging along non stop for three days?
- pimeys 4mo ago
  Fast mode max content window. The task was: replace all 1600+ queries from one database to another and make the whole integration test pass. We did multiple passes, with different concerns when changing from database to another. My OpenCode session right now says $4,365.02. I haven't gotten close to this either before, but now we wanted to move fast because this branch gets conflicts all the time and we want to get over with the migration asap.
  - rglullis 4mo ago
    It's a bit of a left field question, but I am curious: Let's say that if the company wasn't paying the whole bill but only subsidizing it - e.g, if it paid 90% of the $4000. What would you do?
    - 4mo ago

dontlikeyoueith 4mo ago

> OpenAI and Anthropic are heavily subsidizing their inference -- no wait, they are charging the most they can get away with before going public. Where is the truth? Both. They are charging the most they can get away with and that amount is still heavily subsidized by VC capital.

schaefer 4mo ago

> I don't think anyone has a firm grasp on actual inference costs. There are huge numbers of users (myself included) that do have an exact idea of what inference costs are - on open models. We can buy tokens from 3rd parties that have no motivation to subsidize our use. That's to say, there's a fair marketplace[1] and we're hanging out there. If you want to say "I don't think anyone has a firm grasp on actual inference costs on these proprietary/closed models", then I could agree with that. [1]: https://openrouter.ai/rankings#leaderboard https://openrouter.ai/rankings#leaderboard

logicchains 4mo ago

We have a firm grasp on actual inference costs from the various open weights model providers on OpenRouter. They don't have the money to subsidize inference and it's quite a competitive market, so the prices are representative of the costs.

InsideOutSanta 4mo ago

> I don't think anyone has a firm grasp on actual inference costs -- including the research and training that has gone into those models We know roughly how much these companies spend and what their revenues are. Based on that, they'd have to more than double revenue (without spending more money) just to stay even, and that's not good enough given how deep in the hole they are. > OpenAI and Anthropic are heavily subsidizing their inference -- no wait, they are charging the most they can get away with before going public. Where is the truth? Both are true. I mean, I'd be willing to spend a bit more than I do now, but not more than double, and neither are most companies. The company I work for is currently investigating how to reduce LLM spend, not looking to spend more.