16 ms·

Are OpenAI and Anthropic losing money on inference?

• techpineapple 1y ago
  So, if this is true, OpenAI needs much better conversion rates, because they have ~15 million paying users compared to 800 million weekly active users: https://nerdynav.com/chatgpt-statistics/ https://nerdynav.com/chatgpt-statistics/
  • martinald 1y ago
    Yeah but they can probably monetize them with ads.
    • bgwalter 1y ago
      I'm not so sure. Inserting ads into chatbot output is like inserting ads into email. People are more reluctant to tolerate that than web or YouTube ads (which are hated already). If they insert stealth ads, then after the third sponsored bad restaurant suggestion people will stop using that feature, too.
      • martinald 1y ago
        Mmm let's see. I think in LLM ads are probably have the most intent (and therefore most value) of any ads. They are like search PPC ads on steroids as you have even more context of what the user is actually looking for. Hell they could even just add affiliate tracking to links (and not change any of the ranking based on it) and probably make enough money to cover a lot of the inference for free users.
    • UltraSane 1y ago
      LLM generated ads.
• JCM9 1y ago
  These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage payment, not because they’re negative on operating costs. If the cash flow was truly healthy these companies wouldn’t need to raise money. If you have healthy positive cash flow you have much better mechanisms available to fund capital investment other than selling shares at increasingly inflated valuations. Eg issue a bond against that healthy cash flow. Fact remains when all costs are considered these companies are losing money and so long as the lifespan of a model is limited it’s going to stay ugly. Using that apartment building analogy it’s like having to knock down and rebuild the building every 6 months to stay relevant, but saying all is well because the rents cover the cost of garbage collection and the water bill. That’s simply not a viable business model. Update Edit: A lot of commentary below re the R&D and training costs and if it’s fair to exclude that on inference costs or “unit economics.” I’d simply say inference is just selling compute and that should be high margin, which the article concludes it is. The issue behind the growing concerns about a giant AI bubble is if that margin is sufficient to cover the costs of everything else. I’d also say that excluding the cost of the model from “unit economics” calculations doesn’t make business/math/economics since it’s literally the thing being sold. It’s not some bit of fungible equipment or long term capital expense when they become obsolete after a few months. Take away the model and you’re just selling compute so it’s really not a great metric to use to say these companies are OK.
  • politelemon 1y ago
    What will be the knock on effect on us consumers?
    • kelp6063 1y ago
      API prices are going up and rate limits are getting more aggressive (see what's going on with cursor and claude code)
    • 1y ago
• gjsman-1000 1y ago
  If inference is that cheap, why is not even one company profitable yet?
  • ascorbic 1y ago
    Because they're spending it all on training the next model.
    • topaz0 1y ago
      That's an argument for why openai and anthropic shouldn't be profitable, but this point is about how also they don't have customers using the models to generate a profit either. Things like cursor, for example. ETA: also note the recent MIT study that found that 95% of LLM pilots at for-profit companies were not producing returns.
      • ascorbic 1y ago
        This article is about the model providers' costs, not API users'. Cursor etc have to pay the marked-up inference costs, so it's not surprising they can't make a profit.
        • topaz0 1y ago
          Yes, and the comment you first replied to was about the state/viability of the industry as a whole. If users can't make money from this "transformative technology", even when the provider is in the stage of burning money for the sake of growth, that sort of tells against it turning into a trillion dollar industry or whatever the hype claims.
          • ascorbic 1y ago
            The point is that the providers aren't burning money by subsidising inference costs. On the contrary, if this article is to believed they're charging healthy margins on it. So there are two answers: for the model providers, it's because they're spending it all on training the next model. For the API users, it's because they're spending it all on expensive API usage.
• player1234 1y ago
  Input inference i.e. reading is cheaper, output i.e. doing the generating is not, for something called generative AI sounds pretty fucking not profitable. The cheap usecase from this article is not a trillion dollar industry and absolutely not the usecase hyped as the future by AI companies, that is coming for your job.
• jsnell 1y ago
  I don't believe the asymmetry between prefill and decode is that large. If it were, it would make no sense for most of the providers to have separate pricing for prefill with cache hits vs. without. Given the analysis is based on R1, Deepseek's actual in-production numbers seem highly relevant: https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md https://github.com/deepseek-ai/open-infra-index/blob/main/20... (But yes, they claim 80% margins on the compute in that article.) > When established players emphasize massive costs and technical complexity, it discourages competition and investment in alternatives But it's not the established players emphasizing the costs! They're typically saying that inference is profitable. Instead the false claims about high costs and unprofitability are part of the anti-AI crowd's standard talking points.
  • martinald 1y ago
    Yes. I was really surprised at this myself (author here). If you have some better numbers I'm all ears. Even on my lowly 9070XT I get 20x the tok/s input vs output, and I'm not doing batching or anything locally. I think the cache hit vs miss stuff makes sense at >100k tokens where you start getting compute bound.
    • jsnell 1y ago
      I linked to the writeup by Deepseek with their actual numbers from production, and you want "better numbers" than that?! > Each H800 node delivers an average throughput of ~73.7k tokens/s input (including cache hits) during prefilling or ~14.8k tokens/s output during decoding. That's a 5x difference, not 1000x. It also lines up with their pricing, as one would expect. (The decode throughputs they give are roughly equal to yours, but you're claiming a prefill performance 200x times higher than they can achieve.)
      • smarterclayton 1y ago
        A good rule of thumb is that a prefill token is about 1/6th the compute cost of decode token, and that you can get about 15k prefill tokens a second on Llama3 8B on a single H100. Bigger models will require more compute per token, and quantization like FP8 or FP4 will require less.
• osti 1y ago
  This kinda tracks with the latest estimate of power usage of llm inference published by google https://news.ycombinator.com/item?id=44972808 https://news.ycombinator.com/item?id=44972808. If inference isnt that power hungry like people thought, they must be able to make good money from those subscriptions.
  • jeffbee 1y ago
    > power hungry like people thought The only people who thought this were non-practitioners.
• pityJuke 1y ago
  From https://www.theverge.com/command-line-newsletter/759897/sam-altman-chatgpt-openai-social-media-google-chrome-interview https://www.theverge.com/command-line-newsletter/759897/sam-..., Sam Altman said: > “If we didn’t pay for training, we’d be a very profitable company.”
  • tootie 1y ago
    Yeah I've seen the same sentiment from a few others as well. Inference likely is profitable. Training is incredibly expensive and will sometimes not yield positive results.
  • simlevesque 1y ago
    If we ignore the fact that if training was free, everyone would do it and OpenAI wouldn't be profitable.
  • Aurornis 1y ago
    Exactly. All of the claims that OpenAI is losing money on every request are wrong. OpenAI hasn’t even unlocked all of their possible revenue opportunities from the free tier such as ads (like Google search), affiliate links, and other services. There’s also a lot of comments in this thread who want LLM companies to fail for different reasons, so they’re projecting that wish on to imagined unit economics. I’m having flashbacks to all of the conversations about Uber and claims that it was going to collapse as soon as the investment money ran out. Then Uber gradually transitioned to profitability and the critics moved to using the same shtick on AI companies.
    • techpineapple 1y ago
      Because Sam Altman said so? Sam Altman also said this: https://xcancel.com/sama/status/1876104315296968813 https://xcancel.com/sama/status/1876104315296968813
      • JimDabell 1y ago
        There is a six month gap between those statements. Inference costs have been plummeting, plans have had tweaked quotas, and usage patterns can change.
• jonathan-adly 1y ago
  Basically- the same math as modern automated manufacturing. Super expensive and complex build-out - then a money printer once running and optimized. I know there is lots of bearish sentiments here. Lots of people correctly point out that this is not the same math as FAANG products - then they make the jump that it must be bad. But - my guess is these companies end up with margins better than Tesla (modern manufacturer), but less than 80%-90% of "pure" software. Somewhere in the middle, which is still pretty good. Also - once the Nvidia monopoly gets broken, the initial build out becomes a lot cheaper as well.
  • hugedickfounder 1y ago
    the difference is you can train on outputs deepseek style, there are not gates in this field profit margins will go to 0
  • Workaccount2 1y ago
    The difference is the money printer right now only prints for ~6 months before it needs to be replaced with an even more expensive printer.
    • churchill 1y ago
      And if you ever stop/step off the treadmill and jack up prices to reach profitability, a new upstart without your sunk costs will immediately create a 99% solution and start competing with you. Or more like hundreds of competitors. Like we've seen with Karpathy & Murati, any engineer with pedigree working on the frontline models can easily raise billions to compete with them. Expect the trend to pick up as the pool of engineers who can create usable LLMs from scratch increases through knowledge/talent diffusion.
      • Workaccount2 1y ago
        The LLM scene is an insane economic bloodbath right now. The tech aside, the financial moves here are historical. It's the ultimate wet dream for consumers - many competitors, face-ripping cap-ex, any missteps being quickly punished, and a total inability to hold back anything from the market. Companies are spending hundreds of billions to put the best tech in your hands as fast and as cheaply as possible. If OpenAI didn't come along with ChatGPT, we would probably just now be getting Google Bard 1.0 with an ability level of GPT-3.5 and censorship so heavy it would make it useless for anything beyond "Tell me who the first president was".
• JCM9 1y ago
  With the heat turning up on AI companies to explain how they will land on a viable business model some of this is starting to look like WeWork’s “Community Adjusted EBITA” arguments of “hey if you ignore where we’re losing money, we’re not losing money!” that they made right before imploding. I think most folks understand that pure inference in a vacuum is likely cash flow positive, but that’s not why folks are asking increasingly tough questions on the financial health of these enterprises.
  • Aurornis 1y ago
    A fast growing venture backed startup doing frontier R&D should be losing money overall. If they weren’t losing money, they wouldn’t be spending enough on R&D. This isn’t some gotcha. It’s what the investors want right now.
    • JCM9 1y ago
      Don’t disagree it’s what investors want. Point is just that we’re approaching a point from an economics standpoint where the credibility of the “it’s ok because we’re investing in R&D” argument is rapidly wearing thin. WeWork’s investors didn’t want them to focus on business fundamentals either and kept pumping money elsewhere. That didn’t turn out so well.
• gpjanik 1y ago
  "Here's the key insight: each forward pass processes ALL tokens in ALL sequences simultaneously." This sounds incorrect, you only process all tokens once, and later incrementally. It's an auto-regressive model after all.
  • sailingparrot 1y ago
    Not during prefill, i.e. the very first token generated in a new conversation. During this forward pass, all tokens in the context are all processed at the same time, and then attention's KV are cached, you still generate a single token, but you need to compute attention from all tokens to all tokens. From that point on every subsequent tokens is processed sequentially in autoregressive way, but because we have the KV cache, this becomes O(N) (1 token query to all tokens) and not O(N^2)
    • gpjanik 1y ago
      I somehow missed the "decode phase" paragraph and hence was confused - it's essentially that separation I meant, you're obviously correct.
• uludag 1y ago
  Another comment mentioned the cost associated with the model. Setting that aside, wouldn't we also need to include all of the systems around the inference? I can imagine significant infrastructure and engineering needs around all of these various services, along with the work needed to keep these systems up and running. Or are these costs just insignificant compared to inference?
  • Aurornis 1y ago
    All incremental costs should be included. If adding each 100,000 new customers requires 1 extra engineer you would include that. We don’t know those exact numbers though and the ratio is probably much higher than my example numbers. Inference costs likely dominate.
• gitremote 1y ago
  These numbers are off. > $20/month ChatGPT Pro user: Heavy daily usage but token-limited ChatGPT Pro is $200/month and Sam Altman already admitted that OpenAI is losing money from Pro subscriptions in January 2025: "insane thing: we are currently losing money on openai pro subscriptions! people use it much more than we expected." - Sam Altman, January 6, 2025 https://xcancel.com/sama/status/1876104315296968813 https://xcancel.com/sama/status/1876104315296968813
  • mvieira38 1y ago
    Doesn't he have an incentive to make it look like that, though? The way he phrased it, that they are losing money because people use it so much, makes it seem like Pro subscribers are some super power-users. As long as inference has a nonnegative, nonzero cost, then this case will lose money, so Sam isn't admitting that the business model is flawed or anything
  • AstroBen 1y ago
    I just straight up don't trust him Saying that is the equivalent of him saying "our product is really valuable! use it!"
    • benzible 1y ago
      There's the usual issue of a CEO "talking their book" but there's also the fact that Sam has a rich, documented history of lying. That was the central issue of his firing. "Empire of AI" has a detailed account of this. He would outright tell board member A that "board member B said X", based on his knowledge of the social dynamics of the board he assumed that A and B would never talk. But they eventually figured it out, it unraveled, and they confronted him in a group. Specifically, when they confronted him about telling Ilya Sutskever that Tasha McCauley said Helen Toner should step off the board, McCauley said "I never said that" and Altman was at a loss for words for a minute before finally mumbling "Well, I thought you could have said that. I don't know."
    • GoatInGrey 1y ago
      That is my interpretation, that it's a marketing attempt. A form of "The value of our product is so good that it's losing us money. It's practically the Costco hotdog combo!".
• rossdavidh 1y ago
  "Heavy readers - applications that consume massive amounts of context but generate minimal output - operate in an almost free tier for compute costs." Not saying there's not interesting analysis here, but this is assuming that they don't have to pay for access to the massive amounts of context. Sources like stackoverflow and reddit that used to be free, are not going to be available to keep the model up to date. If this analysis is meant to say "they're not going to turn the lights out because of the costs of running", that may be so, but if they cannot afford to keep training new models every so often they will become less relevant over timte, and I don't know if they will get an ocean of VC money to do it all again (at higher cost than last time, because the sources want their cut now).
• EcommerceFlow 1y ago
  I wouldn't be surprised if their profit/query is at a negative for all major Ai companies, but guess what? They have a service which understands a users question/needs 100x better than a traditional Google search does. Once they tap into that for PPC/paid ads, their profit/query should jump into the green. In fact, there's a decent chance a lot of these models will go 100% free once that PPC pipeline is implemented and shown to be profitable.
  • techpineapple 1y ago
    > They have a service which understands a users question/needs 100x better than a traditional Google search does. Source?
    • EcommerceFlow 1y ago
      A lifetime of using Google and 4 years of using LLMs.
      • nativeit 1y ago
        …is a great counter-example of a “source”. It’s not like the product at-hand is relevant to data analysis or anything, amirite?
        • Filligree 1y ago
          Sometimes a statement is just too obvious to need extensive sourcing, and this is one of those times. Gemini doesn’t always find very much better results, but it usually does. It beggars belief to claim that it doesn’t also understand the query much better than Rankbrain et al.
  • efficax 1y ago
    > Once they tap into that for PPC/paid ads, If they start showing ads based on your prompts, and your history of "chats", it will erode the already shaky trust that users have in the bots. "Hallucinations" are one thing, but now you'll be asking yourself all the time: is that the best answer the llm can give me, or has it been trained to respond in ways favourable to its advertisers?
• OtherShrezzing 1y ago
  This is a great article, but it doesn't appear to model H100 downtime in the $2/hr costs. It assumes that OpenAI and Anthropic can match demand for inference to their supply of H100s perfectly, 24/7, in all regions. Maybe you could argue that the idle H100s are being used for model training - but that's different to the article's argument that inference is economically sustainable in isolation.
  • manquer 1y ago
    Not really, that is why they sell Batch API at considerably lower costs than the normal API. There are also probably all kinds of enterprise deals that they are okay with high latency (> hours) that they do beyond the PAYG batch APIs
• cjbarber 1y ago
  See also: https://x.com/tanayj/status/1960116730786918616 https://x.com/tanayj/status/1960116730786918616 OpenAI projects 50% gross margins for 2025 The other companies don't include free users in their GM calculations which makes it hard to compare
• moduspol 1y ago
  This kind of presumes you're just cranking out inference non-stop 24/7 to get the estimated price, right? Or am I misreading this? In reality, presumably they have to support fast inference even during peak usage times, but then the hardware is still sitting around off of peak times. I guess they can power them off, but that's a significant difference from paying $2/hr for an all-in IaaS provider. I'm also not sure we should expect their costs to just be "in-line with, or cheaper than" what various hourly H100 providers charge. Those providers presumably don't have to run entire datacenters filled to the gills with these specialized GPUs. It may be a lot more expensive to do that than to run a handful of them spread among the same datacenter with your other workloads.
  • GaggiX 1y ago
    That's why they have the batch tier: https://platform.openai.com/docs/guides/batch https://platform.openai.com/docs/guides/batch
  • lolc 1y ago
    Of course it is impossible for us to know the true cost, but idle instances should not be accounted for at full price: 1. Idle instances don't turn electricity to heat so that reduces their operating cost. 2. Idle instances can be borrowed for training which means flexible training amortizes peak inference capacity.
  • martinald 1y ago
    Yes. But these are on demand prices, so you could just turn them off when loads are less. But there is no way that OpenAI should be more expensive than this. The main cost is the capex of the H100s, and if you are buying 100k at a time you should be getting a significant discount off list price.
  • empath75 1y ago
    > In reality, presumably they have to support fast inference even during peak usage times, but then the hardware is still sitting around off of peak times. I guess they can power them off, but that's a significant difference from paying $2/hr for an all-in IaaS provider. They can repurpose those nodes for training when they aren't being used for inference. Or if they're using public cloud nodes, just turn them off.
• ekelsen 1y ago
  The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out. 1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs And that's assuming a more likely 1 byte per parameter. So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on their conclusions...
  • deleted 1y ago
    [deleted]
  • Lionga 1y ago
    Well he asked some AI to do the math for him probably
  • GaggiX 1y ago
    Your calculations make no sense. Why are you loading the model for each token independently? You can process all the input tokens at the same time as long as they can fit in memory. You are doing the calculation as they were output tokens on a single batch, it would not make sense even in the decode phase.
    • ozgung 1y ago
      This. ChatGPT also agrees with you: "74 GB weight read is per pass, not per token." I was checking the math in this blog post with GPT to understand it better and it seems legit for the given assumptions.
    • ekelsen 1y ago
      Then the right calculation is to use FLOPs not bandwidth like they did.
  • endtime 1y ago
    > 37e9 bytes/token This doesn't quite sound right...isn't a token just a few characters?
• smjburton 1y ago
  Good breakdown of the costs involved. Even if they're running at a loss, OpenAI and Anthropic receive considerable value from the free training data users are providing through their conversations. Looking at it another way, these companies are paying for the training data to make their models better for future profitability.
• Mallowram 1y ago
  [flagged]
• caminanteblanco 1y ago
  Ok, one issue I have with this analysis is the breakdown between input and output tokens. I'm the kind of person who spend most of my chat asking questions, so I might only use 20ish input tokens per prompt, where Gemini is having to put out several hundred, which would seem to affect the economics quite a bit
  • bcrosby95 1y ago
    Yeah, I've noticed Chatgpt5 is very chatty. I can ask a 1 sentence question and get back 3-4 paragraphs, most of which I ignore, depending upon the task.
    • ozgung 1y ago
      Same. It acts like its output tokens are for free. My input output ratio is like 1 to 10 at least. Not counting "Thought" and it's internal generation for agentic tasks.
    • trashface 1y ago
      Switch to Robot personality
    • solarkraft 1y ago
      I haven’t used it without customization, but I find it follows my brevity user instructions more strictly.
  • pakitan 1y ago
    It may hurt them financially but they are fighting for market share and I'd argue short answers will drive users away. I prefer the long ones much more as they often include things I haven't directly asked about but are still helpful.
  • red2awn 1y ago
    It also didn't take into account a lot of the new models are reasoning models which spits out a lot of output tokens.
• softwaredoug 1y ago
  I wonder if there needs to be two different business models: 1. Companies that train models and license them 2. Companies that do inference on models
• sc68cal 1y ago
  This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.
  • boroboro4 1y ago
    DeepSeek inference efficiency comes from two things: MoE and MLA attention. OpenAI was rumored to use MoE around GPT4 moment, I.e loooong time ago. Given Gemini efficiency with long context I would bet their attention is very efficient too. GPT OSS uses fp4, which DeepSeek doesn’t use yet btw. So no, big labs aren’t behind DeepSeek in efficiency. Not by much at least.
  • GaggiX 1y ago
    The "efficiency" meantioned in blog post you have linked is the price difference between Deepseek and o1, it doesn't mean that GPT-5 or other SOTA models are less efficient.
  • CjHuber 1y ago
    The reason it shook the market at least was because of the claim that its training cost was 5 million.
    • hirako2000 1y ago
      That' what the buzz focused on, strange as we don't actually know what it cost them. While inference optimization is a fact and is even more impactful since training costs benefit from economics of scale.
      • CjHuber 1y ago
        I don't think that's strange at all, it's a much more palatable narrative for the mass who doesn't know what inference and training is and who think having conversations=training
• teekert 1y ago
  Idk what is going on but I'm using it all day for free, no limits in sight yet... It's just for small things, but for sure I would have had to pay 6 months ago. I actually would if they prompted tbh. Although I still find that whole "You can't use the webUI with your API credits" annoying. Why not? Why make me run OpenWebUI or LibreChat? I guess my use is absolutely nothing compare to someone with a couple of agents running continuously.
• kaelandt 1y ago
  The API prices of $3/$15 are not right for a lot of models. see at openrouter, the gpt-oss-120b ones https://openrouter.ai/openai/gpt-oss-120b https://openrouter.ai/openai/gpt-oss-120b, it's more like $0.01/$0.3 (and that model actually needs h200/b200 to have good throughput).
• layer8 1y ago
  The only reason they wouldn’t be losing money on inference is if more costly (more computationally intensive) inference wouldn’t be able to give them an extra edge, which seems unlikely to me.
• noodletheworld 1y ago
  Huh. I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices? The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1? Surely if its this cheap, and we're talking massive margins according to this, I should be able to get a cheap / run my own 600B param model. Am I missing something? It seems that reality (ie. the absence of people actually doing things this cheap) is the biggest critic of this set of calculations.
  • brokencode 1y ago
    I also have no idea on the numbers. But I do know that these same companies are pouring many billions of dollars into training models, paying very expensive staff, and building out infrastructure. These costs would need to be factored in to come up with the actual profit margins.
  • GaggiX 1y ago
    https://openrouter.ai/deepseek/deepseek-chat-v3.1 https://openrouter.ai/deepseek/deepseek-chat-v3.1 They are dirt cheap. Same model architecture for the comparison: $0.30/M $1.00/M. Or even $0.20-$0.80 from another provider.
  • hirako2000 1y ago
    Imo the article is totally off the mark since it assumes users on average do not go over th 1M tokens per day. Afaik openai doesn't enforce a daily quota even on the $20 plans unless the platform is under pressure. Since I often consume 20M token per day, one can assume many would use far more than the 1M tokens assumed in the article's calculations.
    • empath75 1y ago
      There's zero basis for assuming any of that. The most likely situation is a power law curve where the vast majority of users don't use it much at all and the top 10% of users account for 90% of the usage. It is very likely that you are in the top 10% of users.
• ath3nd 1y ago
  Yes they are, they are deeply deeply unprofitable and that's why they need endless investments to prop them up. That's why Microsoft is not doing the deal with OpenAI, that's why Claude was fiddling with token limits just a couple of weeks ago. It's a huge bubble, and the only winner at this moment is Nvidia.
  • wahnfrieden 1y ago
    Citation? Investments aren't evidence of unprofitability in inference
    • ath3nd 1y ago
      > Citation? Investments aren't evidence of unprofitability in inference Does a quote from their CEO help you? "Anthropic CEO Dario Amodei has indicated that the individual models that the company creates are profitable, even if one includes the cost of training them. But given how Anthropic keeps training new models, it reports overall losses at a company level." https://officechai.com/ai/each-individual-ai-model-can-already-be-profitable-anthropic-ceo-dario-amodei/ https://officechai.com/ai/each-individual-ai-model-can-alrea... That good enough for you? It's also obvious by the fact they are starting to train on your data, like the grifters they are: https://www.theverge.com/anthropic/767507/anthropic-user-data-consumers-ai-models-training-privacy https://www.theverge.com/anthropic/767507/anthropic-user-dat... They are struggling to do anything to survive, with a crushing debt. Soon Ads :)
      • wahnfrieden 1y ago
        I asked about profitability of inference because that’s what this article/thread are about. Your quote is evidence that inference is profitable. Thank you.
        • ath3nd 1y ago
          Haha, profitable on some models is not the same as profitable. They are deeply unprofitable as a company and that's why they are grifting, changing plans and lowering limits, and now training on your data. Grifters gonna grift.
• mutkach 1y ago
  A full KV-cache is quite big compared to the weights of the model (depending on the context size), that should be a factor too (and basically you need to maintain a separate KV cache for each request, I think...). Also the the token/s is not uniform across the request and it's getting slower with each subsequent generated token. On the other side, there's an insane booster of speculative decoding, that would give a semi-prefill rate for decoding, but the memory pressure is still a factor. I would be happy to be corrected regarding both factors.
• SamInTheShell 1y ago
  Those same H100's are probably also going to be responsible for the R&D of new model versions. Running a model is definitely cheaper than training them.
• ankit219 1y ago
  This seems very very far off. From the latest reports, anthropic has a gross margin of 60%. It came out in their latest fundraising story. From that one The Information report, it estimated OpenAI's GM to be 50% including free users. These are gross margins so any amortization or model training cost would likely come after this. Then, today almost every lab uses methods like speculative decoding and caching which reduce the cost and speed up things significantly. The input numbers are far off. The assumption is 37B of active parameters. Sonnet 4 is supposedly a 100B-200B param model. Opus is about 2T params. Both of them (even if we assume MoE) wont have exactly these number of output params. Then there is a cost to hosting and activating params at inference time. (the article kind of assumes it would be the same constant 37B params).
  • mutkach 1y ago
    Gross margins also don't tell the whole story, we don't know how much Azure and Amazon charge for the infrastructure and we have reasons to believe they are selling it at a massive discount (Microsoft definitely does that, as follows from their agreement with OpenAI). They get the model, OpenAI gets discounted infra.
    • ankit219 1y ago
      A discounted Azure H100 will still be more than $2 per hour. Same goes for AWS. Trainium chips are new and not as effective (not saying they are bad) but still cost in the same range. For inference, gross margins are exactly: (what companies charge per 1M tokens to the user) - (direct cost to produce that 1M tokens which is GPU costs).
      • mutkach 1y ago
        I am implying that what OpenAI pays for GPU/hour is much less than $2, because of the discount. That's an assumption. It could be $1, $0.5, no? It could still be burning money for Microsoft/Amazon
  • thegeomaster 1y ago
    Are you saying that you think Sonnet 4 has 100B-200B _active_ params? And that Opus has 2T active? What data are you basing these outlandish assumptions on?
• leecmjohnny 1y ago
  Yes, the API business is cross-subsidizing the consumer business. Back in March, I did the same analysis with greater sensitivities, and arrived at similar gross margins: >70%. https://johnnyclee.com/i/are-frontier-labs-making-80percent-gross-margins-on-llm-tTD9fwOZmuc/ https://johnnyclee.com/i/are-frontier-labs-making-80percent-...
• bjornsing 1y ago
  Can you really rent a cluster of 75 H100s for 75*2 USD per hour? Individual H100s, yes. But with sufficient interconnect to run these huge models?
• deleted 1y ago
  [deleted]
• freediver 1y ago
  Message to Martin if you are reading this - a blog without an RSS feed is not a blog. Please add one :)
• qrios 1y ago
  For sure an interesting calculation. Only one remark from someone with GPU metal experience: > But compute becomes the bottleneck in certain scenarios. With long context sequences, attention computation scales quadratically with sequence length. Even if the statement about quadratically scales is right, the bottleneck we are talking about is somewhere north by factor 1000. If 10k cores do only simple matrix operations each needs to have new data (up to 64k) available every 500 cycles (let's say). Getting these amount of data (without _any_ collision) means something like 100+GByte/s per core. Even 2+TByte/s on HBM means the bottleneck is the memory transfer rate, by something like 500 times. With collision, we talk about an additional factor like 5000 (last time I've done some tests with a 4090).
  • Onavo 1y ago
    What do you mean by collision?
    • agf 1y ago
      I believe it's that the bus can only serve one chip at a time, so it has to actually be faster since sometimes one chip's data will have to wait for the data of another chip to finish first.
    • qrios 1y ago
      If multiple cores tries to get the same memory addresses, the MMU feeds only one core, the second one have to whait. Depends on the type of RAM, this will cost a lot of cycles. GPU MMUs can handle multiple line in parallel. But not 10k cores at the same time. The HBM is not able to transfer 3.5TByte sequencial.
      • whatshisface 1y ago
        Why is that? It seems like multiple cores requesting the same address would be easier for the MMU to fetch for, not harder.
        • qrios 1y ago
• mrcwinn 1y ago
  As the author seems to admit, an outsider is going to lack so much information (costs, loss leaders, etc), one has to assume any modeling is so inaccurate that it's not worth anything. So the question remains unanswered, at least for us. For those putting money in, you can be absolutely certain they have a model with sufficient data to answer the question. Since money did go in, even if it's venture, the answer is probably "yes in the immediate, but no over time."
• alienbaby 1y ago
  Hasn't Sam Altman already said they are profitable on inference, minus training costs?
  • bigdickfounder 1y ago
    [dead]
• simonw 1y ago
  https://www.axios.com/2025/08/15/sam-altman-gpt5-launch-chatgpt-future https://www.axios.com/2025/08/15/sam-altman-gpt5-launch-chat... quotes Sam Altman saying: > Most of what we're building out at this point is the inference [...] We're profitable on inference. If we didn't pay for training, we'd be a very profitable company.
  • drob518 1y ago
    Which is like saying, “If all we did is charge people money and didn’t have any COGS, we’d be a very profitable company.” That’s a truism of every business and therefore basically meaningless.
    • deleted 1y ago
      [deleted]
    • dcre 1y ago
      The Amodei quote in my other reply explains why this is wrong. The point is not to compare the training of the current model to inference on the current model. The thing that makes them lose so much money is that they are training the next model while making back their training cost on the current model. So it's not COGS at all.
      • prasadjoglekar 1y ago
        Well, only if the one training model continued to function as a going business. Their amortization window for the training cost is 2 months or so. They can't just keep that up and collect $. They have to build the next model, or else people will go to someone else.
        • dcre 1y ago
          Why two months? It was almost a year between Claude 3.5 and 4. (Not sure how much it costs to go from 3.5 to 3.7.)
• fancyfredbot 1y ago
  When you are operating at scale you are likely to use a small model during the auto regressive phase to generate sequential tokens and only involve the large model once you've generated several tokens. Whenever the two predict the same output you effectively generate more than one token at a time. The idea is the models will agree often enough to significantly reduce output token costs. Does anyone know how effective that is in practice?
• overgard 1y ago
  I thought the thing that made DeepSeek interesting (besides competition from China) was that its inference costs were something like 1/10th. So unless that gap has been bridged (has it?) I don't think a calculation based on DeepSeek can apply to OpenAI or Anthropic.
• KallDrexx 1y ago
  Since DeepSeek R1 is open weight, wouldn't it be better to validate the napkin math to validate how many realistic LLM full inferences can be done on a single H100 in a time period, and calculate the token cost of that? Without having in depth knowledge of the industry, the margin difference between input and output tokens is very odd to me between your napkin math and the R1 prices. That's very important as any reasoning model explodes reasoning tokens, which means you'll encounter a lot more output tokens for fewer input tokens, and that's going to heavily cut into the high margin ("essentially free") input token cost profit. Unless I'm reading the article wrong.
  • RhythmFox 1y ago
    I am so glad someone else called this out, I was reading the napkin math portions and struggling to see how the numbers really worked out and I think you hit the nail on the head. The author is assuming 'essentially free' input token cost and extrapolating in a business model that doesn't seem to connect directly to any claimed 'usefulness'. I think the bias on this is stated in the beginning of the article clearly as the author assumes 'given how useful the current models are...'. That is not a very scientific starting point and I think it leads to reasoning errors within the business model he posits here. There were some oddities with the numbers themselves as well but I think it was all within rounding, though it would have been nice for the author to spell it out when he rounded some important numbers (~s don't tell me a whole lot). TL;DR I totally agree, there are some napkin math issues going on here that make this pretty hard to see as a very useful stress test of cost.
• techlatest_net 1y ago
  [dead]
• stephenatgmi 1y ago
  Some agent startups are already feeling the squeeze — The Information reported Cursor’s gross margins hit –16% due to token costs. So even if inference is profitable for OAI/Anthropic, downstream token-hungry apps may not see the same unit economics, and that is why token-intensive agent startups like Cursor and Perplexity are taking open-source models like Qwen or other OSS-120B and post-training them to bring down inference costs.
• snowwrestler 1y ago
  Not wishing to do a shallow dismissal here, but I always assumed AI must be profitable on inference otherwise no one would pursue it as a business given how expensive the training is. It seems sort of like wondering if a fiber ISP is profitable per GB bandwidth. Of course it is; the expensive part is getting the fiber to all the homes. So the operations must be profitable or there is simply no business model possible.
  • overgard 1y ago
    AI right now seems more like a religious movement than a business one. It doesn't matter how much it costs (to the true believers), its about getting to AGI first.
• _sword 1y ago
  I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to be considered... yikes
  • lawlessone 1y ago
    Does that include legal fights and potential payouts to artists and writers whose work was used without permission? Can anyone explain why it's not allowed to compensate the creators of the data?
    • ergocoder 1y ago
      Of course not. Those usually wouldn't be considered "margin". Another similar example is R&D and development by engineers aren't considered in margin either.
    • Night_Thastus 1y ago
      It's already questionable if anyone can make it profitable once you account for all the costs. Why do you think they try to squash the legal concerns so hard? If they move fast and stick their fingers in their ears, they can just steal whatever the want.
    • datavirtue 1y ago
      [flagged]
      • lawlessone 1y ago
        why not?
        • eikenberry 1y ago
          Obviously because you are now allowed to download and share copyrighted works without permission or cost. At least that seems to be the precedent being set in court cases thus far.
• WhitneyLand 1y ago
  Model context limits are not “artificial” as claimed. The largest context window a model can offer at a given quality level depends on the context size the model was pretrained with as well as specific fine tuning techniques. It’s not simply a matter of considering increased costs.
  • Der_Einzige 1y ago
    Context extension methods exist and work. Please educate yourself about these rather than confidentially saying wrong things.
    • WhitneyLand 1y ago
      Not sure what you’re disagreeing with? Context window size limits are not artificial. It takes real time/money/resources to increase them. There are a few ways to approach the problem. Pre-training on longer context lengths I’ve already mentioned. Fine-tuning techniques (like LongRoPE) I’ve already mentioned. Inference time context extension tricks I didn’t mention because the papers I’ve seen seem to suggest there’s often problems with quality or unfavorable tradeoffs. There’s no magic way around these limits, it’s a real engineering problem.
• resters 1y ago
  Consider some of the scaling properties of frontier cloud LLMs: 1) routing: traffic can be routed to smaller, specialized, or quantized models 2) GPU throughput vs latency: both parameters can be tuned and adjusted based on demand. What seems like lots of deep "thinking" might just be trickling the inference over less GPU resources for longer. 3) caching
• fallmonkey 1y ago
  The estimation for output token is too low since one reasoning-enabled response can burn through thousands of output tokens. Also low for input tokens since in actual use there're many context (memory, agents.md, rules, etc) included nowadays.
  • atq2119 1y ago
    When using APIs, you pay for reasoning tokens like you do for actual outputs. So, the estimation on a per-token basis is not affected by reasoning. What reasoning affects is the ratio of input to output tokens, and since input tokens are cheaper, that may well affect the economics in the end.
    • fallmonkey 1y ago
      Correct, and with reasoning, the ratio is totally off. As others have pointed out, actual usage is way higher (much more than 3-5x) than the estimation in the article, which is probably for very trivial users.
• reenorap 1y ago
  An interesting exercise would be what prompts would create the most costs for LLMs but outwards little to no costs for the issuers. Are all prompts equal and the only factor being the lengths of the input and output prompts? Or is there processing of the prompts that could be exceedingly expensive for the LLM?
• deleted 1y ago
  [deleted]
• ProofHouse 1y ago
  Only introducing this *NVIDIA AI Released Jet-Nemotron: 53x Faster Hybrid-Architecture Language Model Series that Translates to a 98% Cost Reduction for Inference at Scale, into the conversation as it just dropped (so it is timely) and while it seems unlikely either OpenAI or Anthropic use this or a technique like it (yet or if they even can), these types of breakthroughs may introduce dramatic savings for both closed and open source inference at scale moving forward https://www.marktechpost.com/2025/08/26/nvidia-ai-released-jet-nemotron-53x-faster-hybrid-architecture-language-model-series-that-translates-to-a-98-cost-reduction-for-inference-at-scale/ https://www.marktechpost.com/2025/08/26/nvidia-ai-released-j...
• atleastoptimal 1y ago
  Everyone claiming AI companies are a financial ticking time bomb are using the same logic people used back in the 2000s when they claimed Amazon “never made a profit” and thus was a bad investment.
  • 9cb14c1ec0 1y ago
    Wrong. The depreciation cost on the hundreds of billions of dollars spent on the AI build-out is almost certainly larger then the AI industry's gross income. This is a vastly different depreciation cost schedule than AWS.
    • atleastoptimal 1y ago
      Well it all seems like a waste of money if you don't believe that AI models are getting significantly more useful every generation, and that the scale-up is on a path to AGI within the decade. These companies obviously do believe in near-term AGI thus even extreme capital expenditure is worthwhile.
• deleted 1y ago
  [deleted]
• yalogin 1y ago
  Will these companies ever stop training new models? What does it mean if we get there. Feels like they will have to constantly train and improve the models, not sure what that means either. What ncremental improvements can these models show? Another question is - will it ever become less costly to train? Let to see opinions from someone in the know
  • Romario77 1y ago
    current way the models works is that they don't have memory, it's included in training (or has to be provided as context). So to keep up with times the models have to be constantly trained. One thing though is that right now it's not just incremental training, the whole thing gets updated - multiple parameters and how the model is trained is different. This might not be the case in the future where the training could become more efficient and switch to incremental updates where you don't have to re-feed all the training data but only the new things. I am simplifying here for brevity, but I think the gist is still there.
    • senko 1y ago
      Updating the internal knowledge is not the primary motivator here, as you can easily, and more reliably (less hallucination), get that information at inference stage (through web search tool). They're training new models because the (software) technology keeps improving, (proprietary) data sets keep improving (through a lot of manual labelling but also synthetic data generation), and in general researchers have better understanding of what's important when it comes to LLMs.
    • yalogin 1y ago
      Sure the training can be made efficient, but how much better can these LLMs get in functionality?
  • jillesvangurp 1y ago
    There's a difference between model training and model fine tuning. Model training is not something e.g. OpenAI does a lot. The cutoff for gpt 5 was actually quite long a go. They sat on that for almost a year while they fine tuned the model. Finetuning is a lot faster and cheaper than training the full model. Training + finetuning is essentially fixed cost. And you have to see this in proportion to the revenue. If you charge 20$/month and you have a few tens of millions of paying users and some premium tier users, that generates quite a bit of revenue. OpenAI recently claimed they have 700 million regular users. I'm not sure how real/accurate that number is, but if one tenth of those pay for it it, that would be 1.4 billion per month coming in. That's excluding higher tiers. And I suspect they are shooting for a much larger market and are going to be nudging people to a bit higher tiers. Some have suggested that employers paying hundreds of dollars per month for AI subscriptions per employee might become normal in some sectors. That's an awful lot of money and I don't think they are done growing. And of course with that kind of revenue, you can burn some cash on training cost. A few hundred million is basically nothing. OpenAI has raised tens of billions of money. But they should be making 10-20 billions of revenue per year as well with some healthy growth. And they are showing very little signs of running out of money.
• chillee 1y ago
  This article's math is wrong on many fundamental levels. One of the most obvious ones is that prefill is nowhere near bandwidth bound. If you compute out the MFU the author gets it's 1.44 million input tokens per second * 37 billion active params * 2 (FMA) / 8 [GPUs per instance] = 13 Petaflops per second. That's approximately 7x absolutely peak FLOPS on the hardware. Obviously, that's impossible. There's many other issues with this article, such as assuming only 32 concurrent requests(?), only 8 GPUs per instance as opposed to the more efficient/standard prefill-decode disagg setups, assuming that attention computation is the main thing that makes models compute-bound, etc. It's a bit of an indictment of HN's understanding of LLMs that most people are bringing up issues with the article that aren't any of the fundamental misunderstandings here.
  • deleted 1y ago
    [deleted]
  • Den_VR 1y ago
    So, bottom line, do you think it’s probable that either OpenAI or Anthropic are “losing money on inference?”
    • chillee 1y ago
      No. In some sense, the article comes to the right conclusion haha. But it's probably >100x off on its central premise about output tokens costing more than input.
      • doctorpangloss 1y ago
        I’m pretty sure input tokens are cheap because they want to ingest the data for training later no? They want huge contexts to slice up.
        • awwaiid 1y ago
          Afaik all the large providers flipped the default to contractually NOT train on your data. So no, training data context size is not a factor.
• sanxiyn 1y ago
  While the broad point that no, OpenAI and Anthropic are not losing money on inference, is almost certainly correct, the model itself is quite incomplete. If you actually want to know, I recommend Inference economics of language models from Epoch AI, which is probably the best public model as of 2025-06. https://arxiv.org/abs/2506.04645 https://arxiv.org/abs/2506.04645
• smcleod 1y ago
  A few things: 1. Your token count per day seems quite low ("2M input tokens, ~30k output tokens/day") - that's FAR less than I'd expect,, for comparison I average 330M - 850M combined tokens per day, I'm on the higher side of my peers that average 150M-600M combined tokens per day. 2. It doesn't seem you're taking prompt caching into account. This generally reduces the inference required for agentic coding by 85-95%. 3. It would be good if you added what quantisation you're running, for example 8.5-9bpw / (Q8 equivalent) (indistinguishable from fp32/bf16) for the model, and for the KV cache (Q8/(b)f16 etc..).
• mcintyre1994 1y ago
  On the latest episode of the Hard Fork podcast, they mentioned they’d just been at an on the record dinner with OpenAI. According to them, Sam said they’re profitable if you exclude the cost of training new models, but his CFO corrected him and said they’re close to that.
• lukev 1y ago
  Things are moving too fast to meaningfully talk about unit economics here. My consumer-level MacBook Pro can run inference locally on models that would have been state-of-the-art anywhere just a little over a year ago. Yet, hosted inference is definitely useful and also can benefit from economies of scale. Also: model training is expensive and the economic reasons for doing so are complex (otherwise why release open weights?) This is an high dimensional space, and unit economics are predicated on reducing to a few meaningful dimensions.
  • deleted 1y ago
    [deleted]
• a333999 1y ago
  Why is everyone so mean? I believe the article delivered on its headline. The author showed that no -large llm providers do not loose money on inference. Model training is not accounted for because that was not the point. I personally felt that the maths calculations was a bit redundant, since after the maths part the same numbers are taken from open router pricing. But I think it is a matter of presentation. I would have shown OR pricing first and then did the math. In that way it would have been as insightful, since it still showed that model providers do also make money and the reader would not have felt that he did maths he could have avoided :) So, thanks to the author! Good job. The feedback fro hn is overly harsh. Idk; It makes me sad how mean people have become. I guess the world is not in a great spot, but taking out anger on strangers will only make it worse.
• johnnienaked 1y ago
  But there's no way those "costs" are real because companies are currently losing money on purpose to accumulate market share and stay ahead of their competitors, similar to Amazon selling diapers at a loss to put diapers.com out of business and then jacking the prices. So the true cost could be 10x as much as stated, we have no idea.