4 ms·

Lights on = Ads in your output. EOY latest; they can't keep kicking the massive costs down the road.

by 1970-01-01 7mo ago

fooster 7mo ago
Where is your evidence of this "massive cost"? Inference is massively profitable for both anthropic and openai. Training is not.
- wesammikhail 7mo ago
  source?
  - KaoruAoiShiho 7mo ago
    After googling https://www.reddit.com/r/singularity/comments/1psesym/openais_compute_margin_said_to_jump_to_70/ https://www.reddit.com/r/singularity/comments/1psesym/openai...
    - caminante 7mo ago
      >OpenAI's compute margin, referring to the share of revenue excluding the costs of running its AI models for paying users Huh? The reddit summary comment makes no sense. How are they getting revenues without ads or paying customers? "After" makes more sense. FTA: >The company has yet to show a profit and is searching for ways to make money to cover its high computing costs and infrastructure plans.
    - wesammikhail 7mo ago
      I've seen sources like this before. It's all hearsay and promo. I was asking for any publicly available verifiable information regarding the cost of inference at scale. I haven't seen any such info personally which is why I asked. I'm dying to see S-1 filing for Anthropic or OpenAI. I don't actually think inference is as cheap as people say if you consider the total cost (hardware, energy, capex, etc)
      - KaoruAoiShiho 7mo ago
        Well they're not public yet so you'll have to put up with rumors. But the numbers are available for companies like DeepSeek say they have an 80% profit margin, so it stands to reason OAI etc would do similar numbers considering they charge much more.
        caminante 7mo ago
        AFAIK, 1. the 80% margin from 2025 was theoretical, 2. they're relying on distillation/synthetic data for training, 3. and have been very opaque about cross-subsidization of R&D with their models. The distillation alone adds a big asterisk for comparisons.
        KaoruAoiShiho 7mo ago
        Talking nonsense.
        caminante 7mo ago
        > you'll have to put up with rumors. > But the numbers are available for companies like DeepSeek You'd rather trust self-reported figures? LMAO
- scrollop 7mo ago
  The majority of accounts are free - these are profitable? IMO they need as many users before their IPO - then the changes will really begin.
- kibwen 7mo ago
  The evidence is that quotas exist, as seen here, and are low enough that people are hitting them regularly. When was the last time you hit your quota of Google searches? When was the last time you hit your quota of StackOverflow questions? When was the last time you hit your quota of YouTube videos? Any service will rate limit abuse, but if abuse is indistinguishable from regular use from the provider's perspective, that's not a good sign.
  - jerf 7mo ago
    It's also kind of interesting that they don't think they can do what an economy would normally do in this situation, which is raise prices until supply matches. Shortages generally imply mispricing. There's a lot of angles you take from that as a starting point and I'm not confident that I fully understand it, so I'll leave it to the reader.
    - 8note 7mo ago
      the sales pitch is that you can keep throwing more and more tokens at a problem to solve it. if the prices dont keep going down, the pitch falls apart, that you need a specialist to come in and make it work
  - caminante 7mo ago
    Great point. The parent's argument is that the marginal cost of inference is minimal. However, the fundamental flaw is that he's separating inference from the high cost frontier models. It's a cross-subsidy that can't be ignored.
    - bachmeier 7mo ago
      Without any insider knowledge on the economics of these companies, I suspect it's that the amount of infrastructure you have to build is determined by peak usage rather than average usage. If peak usage is much higher for a small part of one day a week (say on Monday morning as software developers across the US get back to work) the cost of fulfilling demand at all times can be insane. That's why companies are implementing batch/standard/priority pricing for the API.
- weakfish 7mo ago
  This article convinced me otherwise https://www.wheresyoured.at/the-subprime-ai-crisis-is-here/ https://www.wheresyoured.at/the-subprime-ai-crisis-is-here/
  - Narciss 7mo ago
    This is a great article, thanks for sharing
  - byzantinegene 7mo ago
    good article!
- quikoa 7mo ago
  Inference for API or subscriptions? There is a massive price difference between the two.
- ares623 7mo ago
  You're assuming they can just stop training. For the entirety of these companies' existence, they have done training. It is part of their price. They must keep pushing out better and better models. That's like saying Nvidia can just stop making new GPUs, they're obviously making so much money with their current models now.
- reppap 7mo ago
  Inference cannot happen without training the model first, so the distinction is quite pointless.
jerf 7mo ago
Ads do not pay enough to cover AI usage. People see the big numbers Google and Facebook make in ads and forget to divide the number by the number of people they serve ads to, let alone the number of ads they served to get to that per-user number. You can't pay for 3 cents of inference with .07 cents of revenue. You also can't put ads in code completion AIs because the instant you do the utility to me of them at work drops to negative. Guess how much money companies are going to pay for negative-value AIs? Let's just say it won't exactly pay for the AI bubble. A code agent AI puts an ad for, well, anything and the AI accidentally puts it into code that gets served out to a customer and someone's going to sue. The merits of the case won't matter, nor the fact the customer "should have caught it in review", the lawsuit and public reputation hit (how many people here are reading this and salivating at the thought of being able to post an angrygram about AIs being nothing but ad machines?) still cost way too much for the AI companies creating the agents to risk.
- willio58 7mo ago
  Agreed, and the answer is pretty obvious as to how they start making profit. The answer is in this thread, CRANKING the cost up immensely once they establish agreements between the duopoly leaders in the field to do so in tandem and buy up any competition that seeks to challenge them. I’m thinking 20x what the cost is now is where they’ll land. It’ll be a massive line item for software dev shops.
  - byzantinegene 7mo ago
    the problem now is how much can they hike the costs without people just going back to coding by hand.
    - einsteinx2 7mo ago
      Or switch to using the way cheaper open weight models from various providers who don’t have to subsidize training costs so can just race to the bottom on inference pricing… The quality isn’t really SOTA yet but at some point I assume they’ll be good enough (maybe already are?).
downrightmike 7mo ago
Pouring 99% of your funds into massive datacenters you can't actually build and ignore product seems short sighted.

Lights on = Ads in your output. EOY latest; they can't keep kicking the massive costs down the road.

Where is your evidence of this "massive cost"? Inference is massively profitable for both anthropic and openai. Training is not.

wesammikhail 7mo ago
source?
- KaoruAoiShiho 7mo ago
  After googling https://www.reddit.com/r/singularity/comments/1psesym/openais_compute_margin_said_to_jump_to_70/ https://www.reddit.com/r/singularity/comments/1psesym/openai...
  - caminante 7mo ago
    >OpenAI's compute margin, referring to the share of revenue excluding the costs of running its AI models for paying users Huh? The reddit summary comment makes no sense. How are they getting revenues without ads or paying customers? "After" makes more sense. FTA: >The company has yet to show a profit and is searching for ways to make money to cover its high computing costs and infrastructure plans.
  - wesammikhail 7mo ago
    I've seen sources like this before. It's all hearsay and promo. I was asking for any publicly available verifiable information regarding the cost of inference at scale. I haven't seen any such info personally which is why I asked. I'm dying to see S-1 filing for Anthropic or OpenAI. I don't actually think inference is as cheap as people say if you consider the total cost (hardware, energy, capex, etc)
    - KaoruAoiShiho 7mo ago
      Well they're not public yet so you'll have to put up with rumors. But the numbers are available for companies like DeepSeek say they have an 80% profit margin, so it stands to reason OAI etc would do similar numbers considering they charge much more.
      - caminante 7mo ago
        AFAIK, 1. the 80% margin from 2025 was theoretical, 2. they're relying on distillation/synthetic data for training, 3. and have been very opaque about cross-subsidization of R&D with their models. The distillation alone adds a big asterisk for comparisons.
        KaoruAoiShiho 7mo ago
        Talking nonsense.
        caminante 7mo ago
        > you'll have to put up with rumors. > But the numbers are available for companies like DeepSeek You'd rather trust self-reported figures? LMAO
scrollop 7mo ago
The majority of accounts are free - these are profitable? IMO they need as many users before their IPO - then the changes will really begin.
kibwen 7mo ago
The evidence is that quotas exist, as seen here, and are low enough that people are hitting them regularly. When was the last time you hit your quota of Google searches? When was the last time you hit your quota of StackOverflow questions? When was the last time you hit your quota of YouTube videos? Any service will rate limit abuse, but if abuse is indistinguishable from regular use from the provider's perspective, that's not a good sign.
- jerf 7mo ago
  It's also kind of interesting that they don't think they can do what an economy would normally do in this situation, which is raise prices until supply matches. Shortages generally imply mispricing. There's a lot of angles you take from that as a starting point and I'm not confident that I fully understand it, so I'll leave it to the reader.
  - 8note 7mo ago
    the sales pitch is that you can keep throwing more and more tokens at a problem to solve it. if the prices dont keep going down, the pitch falls apart, that you need a specialist to come in and make it work
- caminante 7mo ago
  Great point. The parent's argument is that the marginal cost of inference is minimal. However, the fundamental flaw is that he's separating inference from the high cost frontier models. It's a cross-subsidy that can't be ignored.
  - bachmeier 7mo ago
    Without any insider knowledge on the economics of these companies, I suspect it's that the amount of infrastructure you have to build is determined by peak usage rather than average usage. If peak usage is much higher for a small part of one day a week (say on Monday morning as software developers across the US get back to work) the cost of fulfilling demand at all times can be insane. That's why companies are implementing batch/standard/priority pricing for the API.
weakfish 7mo ago
This article convinced me otherwise https://www.wheresyoured.at/the-subprime-ai-crisis-is-here/ https://www.wheresyoured.at/the-subprime-ai-crisis-is-here/
- Narciss 7mo ago
  This is a great article, thanks for sharing
- byzantinegene 7mo ago
  good article!
quikoa 7mo ago
Inference for API or subscriptions? There is a massive price difference between the two.
ares623 7mo ago
You're assuming they can just stop training. For the entirety of these companies' existence, they have done training. It is part of their price. They must keep pushing out better and better models. That's like saying Nvidia can just stop making new GPUs, they're obviously making so much money with their current models now.
reppap 7mo ago
Inference cannot happen without training the model first, so the distinction is quite pointless.

jerf 7mo ago

Ads do not pay enough to cover AI usage. People see the big numbers Google and Facebook make in ads and forget to divide the number by the number of people they serve ads to, let alone the number of ads they served to get to that per-user number. You can't pay for 3 cents of inference with .07 cents of revenue. You also can't put ads in code completion AIs because the instant you do the utility to me of them at work drops to negative. Guess how much money companies are going to pay for negative-value AIs? Let's just say it won't exactly pay for the AI bubble. A code agent AI puts an ad for, well, anything and the AI accidentally puts it into code that gets served out to a customer and someone's going to sue. The merits of the case won't matter, nor the fact the customer "should have caught it in review", the lawsuit and public reputation hit (how many people here are reading this and salivating at the thought of being able to post an angrygram about AIs being nothing but ad machines?) still cost way too much for the AI companies creating the agents to risk.

willio58 7mo ago
Agreed, and the answer is pretty obvious as to how they start making profit. The answer is in this thread, CRANKING the cost up immensely once they establish agreements between the duopoly leaders in the field to do so in tandem and buy up any competition that seeks to challenge them. I’m thinking 20x what the cost is now is where they’ll land. It’ll be a massive line item for software dev shops.
- byzantinegene 7mo ago
  the problem now is how much can they hike the costs without people just going back to coding by hand.
  - einsteinx2 7mo ago
    Or switch to using the way cheaper open weight models from various providers who don’t have to subsidize training costs so can just race to the bottom on inference pricing… The quality isn’t really SOTA yet but at some point I assume they’ll be good enough (maybe already are?).

downrightmike 7mo ago

Pouring 99% of your funds into massive datacenters you can't actually build and ignore product seems short sighted.