5 ms·
It’s incredible how polarizing the AI rush is. I keep the perspective that the technology is an absolute step change but I have no idea where the cards will fal
by infecto 7mo ago
It’s incredible how polarizing the AI rush is. I keep the perspective that the technology is an absolute step change but I have no idea where the cards will fall. I take a lot of issue with these style of articles. I get a sense that the authors are being overly defensive.
The cost to serve tokens is absolutely profitable today and that’s been true for at least a year. What’s unclear is how R&D and capex fit into the picture. I am not that pessimistic on this front either though. For the data center build outs, demand for tokens is still exceeding supply. On the R&D front, well most of us here on HN have benefited from decades of overinflated engineering salaries being paid by often companies that were not profitable and not only unprofitable, usually without a plan for success. In this current rush, companies cannot keep up with supply, it’s a much easier math problem when you have something that people want (tokens) and you need to figure out profitability when including R&D.
- nickphx 7mo agostep change? how? profitable? where did you read that? people want tokens? really? who are these people?
- elzbardico 7mo agoYeah, if we just ignore R&D, fixed costs, depreciation, and the fact that there's a high likelyhood investor were expecting a return, yeah, ignoring all of that, and trusting their number we may say inference turns a profit. In accounting, almost anything you want can be true, at least for some time.
- boriskourt 7mo ago> The cost to serve tokens is absolutely profitable today and that’s been true for at least a year. > For the data center build outs, demand for tokens is still exceeding supply. Can you provide any numbers for this?
- bob1029 7mo agohttps://www.cerebras.ai/blog/cerebras-cs-3-vs-nvidia-dgx-b200-blackwell https://www.cerebras.ai/blog/cerebras-cs-3-vs-nvidia-dgx-b20...
- deleted 7mo ago[deleted]
- paulddraper 7mo agoAnthropic has said inference is profitable. That’s a biased source, but the math pencils. This is why switching to local open weight models saves a lot of money. (Even though it’s not apples to apples.)
- nyeah 7mo agoCan you give a few penciled numbers?
- paulddraper 7mo agoYou can rent a H100 GPU for $4/hour. [1] 300k tokens for that hour. OpenAI charges $6. Those are pessimistic assumptions. [1] https://lambda.ai/instances https://lambda.ai/instances
- drakythe 7mo ago3.99 at 8x instances, with a minimum 2 week commitment. Good luck getting 70% usage average during that time. Useful when you're running a training round and can properly gauge demand, not so great when you're offering an API.
- infecto 7mo agoIs it not a good penciled number? It helps set the directional tone that at inference cost is being covered.
- drakythe 7mo agoIt says the numbers are theoretically possible. Requiring a 66% usage to break even when 100% usage will piss off customers by invoking a queue means it’s a balancing act. “Technically correct. The best kind of correct”. So inference may technically be _capable_ of being profitable, but I have question’s about them being profitable in _practice_.
- 7mo ago
- ACCount37 7mo agoCheck the token prices for open weight LLMs at various independent inference providers. That gives you a very good estimate of "how much can you serve the tokens of a model of the size N for while making a profit". Now, keep in mind: Kimi K2.5 is 1T MoE. Today's frontier LLMs are in the 1T to 5T range, also MoE. Make an estimate. Compare that estimate with the actual frontier lab prices.
- lolc 7mo agoI don't think it's as easy as looking at open weight API prices. We don't know whether the operators are making a profit on all the hardware they bought. Maybe the prices we pay just cover electricity. And it's not even certain that running costs are covered by API prices: The operators may be siphoning content and subsidize from selling that. In the current volatile environment, the API prices are more of a baseline where we can assume it can't be much cheaper to operate these models.
- aurareturn 7mo agoThat doesn't make sense in this environment because everyone is compute constrained with huge backlogs they can't fulfill. If these inference providers aren't making any money, they'd simply sell their GPUs to those who are starved for compute.
- wongarsu 7mo agoI can get Kimi K2.5 inference on openrouter for about $0.5/MTok input + $2.5/MTok output, from six providers that have no moat besides efficiently selling GPU time. We can assume they are doing so at a profit (they have no incentive to do this at a loss), giving us those numbers as the cost to serve a 1T-a32b model at scale. Now we don't know the true size of any of the proprietary models, but my educated guess is that Sonnet is in about the same parameter range, just with better training and much better fine tuning and RLHF. Yet API pricing for Sonnet is $3/MTok input + $15/MTok output, exactly six times as expensive. Even Haiku is twice as expensive as Kimi K2.5. I find it difficult to believe in a world where those API prices aren't profitable. For subscription pricing it's harder to tell. We hear about those that get insane value out of their subscription, but there has to be a large mass who never reaches their limits. With company-wide rollouts there might even be a lot of subscription users who consume virtually no tokens at all.
- jerojero 7mo agoCompanies doing foundational models need to cover the cost of training which is much more expensive than training something like kimi.
- gruez 7mo ago>Companies doing foundational models need to cover the cost of training [...] But that's moving the goalposts? The original claim was on inference itself, not the whole company. > The cost to serve tokens is absolutely profitable today and that’s been true for at least a year.
- lbreakjai 7mo agoBut that's the same as thinking "This bar is selling a cocktail for $15. I could make it at home for 30 cents. They're making $14.7 dollars of profit per cocktail, the owner must be a millionaire now!" Everything is profitable if you ignore the costs.
- wongarsu 7mo agoYes. I would not consider Kimi a particularly good model relative to its size, and making a SotA model is a lot more expensive. But training costs are explicitly excluded when talking about the cost to serve tokens
- infecto 7mo agoMost/all private labs have cited inference is profitable. This was happening before the large push to scrap plans and largely charge folks the underlying api rates. Second take a look at the pricing of open models. Now certainly it’s not direct 1-1 comparison but we can use it as a baseline. Now of course folks might not be telling the truth but one of those situations where I see too many markers on the true side. For supply look at outages and growth rates at companies like openrouter. The demand is growing every week.
- thereitgoes456 7mo ago> The cost to serve tokens is absolutely profitable Can you explain why you know better than the analyst at Cursor cited in this article?
- infecto 7mo agoCan you cite your source of an analyst at Cursor. I read the article and looking through the boatload of links but struggled to find what you are referring to. Ty
- iterateoften 7mo agoOpen router is an upper bound of compute cost for the open source models. So people assume that opus and sonnet really isn’t sucking up 10x the resources because open source models aren’t 10x worse. Idk if it’s true or not, but haiku is $5/m tokens and it is much worse than the $2-3/mt models imo
- vbarrielle 7mo agoOpenrouter is a startup, what's the indication it serves token at a profit? It could be serving them at a loss to show growth.
- noelsusman 7mo agoThat analyst was talking about subsidizing tokens through the subscription plans, which is a different claim.
- infecto 7mo agoTy for sharing and agree. I think there is confusion with some folks in the comments for this post confusing inference profitability and plan profitability. Most plans as we can tell are probably teetering the line of profitability and that’s why we have seen some like Cursor really tighten how many tokens you get.
- malfist 7mo ago> The cost to serve tokens is absolutely profitable today How can you possibly say that? Everyone knows that's not the case, these companies are losing money every day selling tokens. Revenue is not the same thing as profit.
- jeromegv 7mo agoYep, especially if we look at what happened just last week, both Google and Anthropic have dropped how much you get out of your existing plans.
- surajrmal 7mo agoThat's not a necessarily profitability thing as much as a demand thing. The only way to improve the supply for those willing to pay more is to take it away from those paying less. Once supply catches up to demand things will change
- infecto 7mo agoDon’t confuse plan changes with profitability of inference. When people talk about the cost to serve a token and it being profitable we are referring to API cost not the plans which absolutely subsidize some level of use. Hard to know what is breakeven on plan math.
- naravara 7mo agoI think they’re losing money because they have to amortize the costs of training the models in the first place, which is where most of the resource sink is. This is why they were freaking out about DeepSeek just taking the trained model weights and slapping an interface on it.
- malfist 7mo agoThats like saying a restaurant is profitable because they're making money selling meals if you ignore the costs of ingredients. Of course they are profitable if you ignore their cost to bring a product to market.
- SirensOfTitan 7mo agoThis is a classic HN mistaking the map for the territory. R&D and capex absolutely figure into de-facto profitability and sustainability for AI labs, despite their separate treatment in accounting. > well most of us here on HN have benefited from decades of overinflated engineering salaries being paid by often companies that were not profitable and not only unprofitable This is a really concerning perspective: people were paid what they were worth. Software is or was one of the few remaining arenas wherein a person can find a middle or upper middle class lifestyle consistently. I will also note: a startup raising an 8 MM series A and eventually fizzling out is not the same at the hundreds of billions invested in these AI companies without a path to profitability. It is utterly absurd to pretend these are the same thing: any company ingesting that much cash needs to justify its capacity to survive.
- ForHackernews 7mo ago> any company ingesting that much cash needs to justify its capacity to survive. What, why? There are tons of low-margin capex-intensive business out there. I think AI will end up like being like hosting. All the models will converge to being pretty-decent and the companies will have to compete on efficiency since they are selling a generic commodity. You can already see Anthropic fears this scenario since they try so hard to make people use their first-party tools rather than plugging Claude in as a generic part of a third-party stack. LLM hosting is the next VPS.
- nadav_tal 7mo ago[flagged]
- guzfip 7mo ago> Software is or was one of the few remaining arenas wherein a person can find a consistently. I want to add something additional to this: it is one of the few fields that can afford middle or upper middle class lifestyle and is accessible. I have no doubt if I could redo my life with the necessary resources I’d be more than capable of putting myself through med school and gone with a secure career that paid more than I ever made in software. But at this stage of life? I don’t have the time or money to spend a decade+ paying some institution tens of thousands of dollars to hopefully maybe have a real career. Once software as a career dies, I suspect many will find themselves locked out the middle class for generations.
- Aperocky 7mo agoDemand of tokens is absolutely skyrocketing. And unlike the traditional "this will replace humans right away", I think what this introduce is a lot of incentive to spread those token in places where there was never any incentive to hire a software engineer for previously. In turn, that will drive a lot of business activity in those area that will potentially fail given the current quality of the output. This feels like a boom before bust scenario, and I'm not even sure if it will bust.
- hirako2000 7mo agoTulips sales also skyrocketed. Seriously, what value are tokens providing other than justifying layoffs. Concretely. Today. Not in the speculating scenario that cardiologist could be replaced with models. We see this new trend of agentic coding, again a promise software will be written that way going forward, despite the number of fiasco already experienced when trusting a model turned bad. The use case may provide value, but right now all it does is fullfil the push for token consumption all these AI leaders are advocating for.
- Throaway1975123 7mo agoTulips had literally no economic value. LLM's do.
- drakythe 7mo agoI say this as someone who has used them to boilerplate/scaffold a bit of code by this point: Economic Value of LLMs is debatable, if only because they're being too broadly applied.
- Throaway1975123 7mo agoDebatable sure. Not 0. Tulips are 0. They add nothing to anyone's output. LLM's are not. LLM's are not tulips.
- 7mo ago
- Eridrus 7mo agoThe article is just helpfully illustrating how artisanal you can make your slop if you really try!
- deleted 7mo ago[deleted]
- iterateoften 7mo agoAccording to open router token demand is growing at something like 10% a week It’s insane
- infecto 7mo agoI wish this was higher up. I have been tracking the same since Thanksgiving ‘25 and the growth is unreal. Again I don’t know where the cards fall maybe the industry overspent on capex but it’s at least easier to see why they are spending based on demand. The risk of being left out is greater than overbuilding.
- coldpie 7mo agoI do wonder how much of the apparent demand is driven by companies automatically running these things when users didn't actually ask for it. For example every web search I make now has an AI response that I scroll right past. I'm sure that counts for someone's token usage data, but I got zero value from it. This is happening in almost every software product now.
- irke 7mo agoTokens as a metric is the analogue of users as a metric. In the end value per user is what matters in relation to being a healthy going concern and valuation in relation to Meta for example. Value per token is what should matter too - after all that’s what people are paying for.
- chrisweekly 7mo ago> "decades of overinflated engineering salaries" 'Overinflated' relative to what? You make some good points but I don't accept this as a premise.
- schmidtleonard 7mo agoOverinflated relative to the wet dreams of the ownership class.
- gruez 7mo agoIt's not exactly stuff of "wet dreams of the ownership class" to say that of the possible white collar careers, software engineering is pretty hard to beat in terms of salary vs work you need to put in.
- schmidtleonard 7mo agoThis is a story of other careers having salaries pushed down relative to inflating essentials and the resulting economic surplus being squeezed into asset portfolios. It's a story of rich people getting paid for being rich in proportion to how rich they are and soaking up more than 100% of economic growth for the last 50 years. Not a story of software engineers working for a living and getting what would have been a blue-collar salary for it.
- financltravsty 7mo agoCompensation is usually more tightly coupled with leverage, rather than "work."
- mono442 7mo agoAverage salaries for software engineering seem higher compared to other professions because the jobs are mostly in the most expensive to live cities. There's no swe jobs in smaller towns but they're jobs for e.g. accountants.
- fcarraldo 7mo agoWell, not GP, but I do. Let’s look at the numbers: Median senior SWE salaries in SF: https://www.levels.fyi/t/software-engineer/levels/senior/locations/san-francisco-bay-area https://www.levels.fyi/t/software-engineer/levels/senior/loc... Median income in metro areas: https://www.cnbc.com/2024/07/11/the-median-salary-for-the-25-biggest-us-citiessee-how-you-compare.html https://www.cnbc.com/2024/07/11/the-median-salary-for-the-25... Engineering salaries are significantly higher than nearly every other industry on average and on median. Much of this is driven by VC funding rather than sound, profitable, bootstrapped businesses with sustainable profit margins. Engineering salaries have also been driven upwards significantly the past ~10 years (since the post-2008 crash recovery), while wage growth in the US is mostly stagnant. I don’t have a source handy for that, but there are plentiful studies. Outside of the US this may be less true, but I took GP’s “most of us on HN” to mean people who work in US tech companies which are primarily concentrated in high COI areas.
- Tade0 7mo agoMy main worry is - once this is all over, the market consolidates and using LLMs will become a requirement in job listings, what's the highest price per million tokens companies will be able to charge us? Currently on a given day I'm chewing through approximately the equivalent of my lunch money, but where there's opportunity to extract wealth, someone will find a way to do it.
- dist-epoch 7mo agoJensen is already talking about $1000/mil tokens soon. But there is no real higher limit. Imagine a LLM which could answer the question "what does my company need to do to beat the competition?". And then realize that the competition asks their LLM the same question. So now everybody is bidding the price up or using more tokens to get a better answer
- irke 7mo agoComplete nonsense. In that world there’s no reason for a business enterprise to exist.
- Tade0 7mo agoThis is the kind of bullshit I'm worried about the most. Nvidia has a de facto monopoly in the datacenter-tier GPU market. He says this sort of stuff because he knows he can keep jacking up prices, because the cost will be transferred onto consumers - mainly software engineers.
- dist-epoch 7mo agoIf those software engineers keep buying it must mean they get more value out of it than they pay, right?
- h14h 7mo agoMy (potentially naive) take is that open models will save us. The biggest markets for LLMs (e.g. coding) are narrow-enough to be served well by smaller models with proper RL. Cursor's Composer 2 (created from a Kimi K2.5 base) is a great example, and I expect it to be the first of many. The wealth of great open models provide an excellent base for fine-tuning, distillation, and RL. I see a lot of untapped potential in the field of bespoke, purpose-built models that can be served far more cheaply than the frontier competition. I would not be surprised if we see frontier-adjacent experiences running comfortably on a Mac Mini by year end. With frontier models seemingly hitting diminishing returns in quality, I struggle to see a world in which gigantic, expensive, general-purpose models don't become increasingly niche.