4 ms·
If you want LLMs to continue to be offered we have to get to a point where the providers are taking in more money than they are spending hosting them. And we st
by bandrami 6mo ago
If you want LLMs to continue to be offered we have to get to a point where the providers are taking in more money than they are spending hosting them. And we still aren't there (or even close).
- carefree-bob 6mo agoI think this has to be done with technological advances that makes things cheaper, not charging more. I understand why they have to charge more, but not many are gonna be able to afford even $100 a month, and that doesn't seem to be sufficient. It has to come with some combination of better algorithms or better hardware.
- bandrami 6mo agoMaking it more affordable would be very bad news for Amazon, who are now counting on $100B in new spending from OpenAI over the next 10 years.
- philipwhiuk 6mo agoSomeone's going to get burned here that's for sure. This isn't going to end with every person on the planet paying $100 a month for an LLM.
- LtWorf 6mo agoA guy from Meta interviewing at BBC a few years ago claimed that every school child in India was going to have the metaverse VR or they'd be left behind in their education, so every family was certainly going to pony up the money.
- throwthrowuknow 6mo agoSomethings not adding up. Why is Amazon making financial plans for the next decade based on continued OpenAI spending but you’re saying AI providers like OpenAI and Anthropic aren’t even close to being profitable, so how can they last a decade or more? Who’s wrong?
- bandrami 6mo agoI take it you don't remember 2008
- arcanemachiner 6mo agoAre we before or after the part where they start throwing money out of helicopters?
- bandrami 6mo agoThat's the interesting question, right? Because if this unwinds during a period of external inflation (say, because of a big war and energy shortage) then even the Bernanke would say helicopter money won't work
- nimchimpsky 6mo ago[dead]
- Gigachad 6mo agoThey probably aren’t planning on making the money on consumer subscriptions. Any price is viable as long as the user can get more value out of it than they spend.
- bandrami 6mo ago"Sell this for less than it cost us" was a viable business plan during the ZIRP era but is not now
- lynx97 6mo agoI see the current situation as a plus. I get SOTA models for dumping prices. And once the public providers go up with their pricing, I will be able to switch to local AI because open models have improved so much.
- nimchimpsky 6mo ago[dead]
- quikoa 6mo agoThe open models may not be as great but maybe these are good enough. AI users can switch when the prices rise before it becomes sustainable for (some) of the large LLM providers.
- Gigachad 6mo agoCurrently it costs so much more to host an open model than it costs to subscribe to a much better hosted model. Which suggests it’s being massively subsidised still.
- JumpCrisscross 6mo agoIf I drop $10k on a souped-up Mac Studio, can that run a competent open-source model for OpenClaw?
- pbronez 6mo agoRapid MLX team has done some interesting benchmarking that suggests Qwopus 27B is pretty solid. Their tool includes benchmarking features so you can evaluate your own setup. They have a metric called Model-Harness Index: MHI = 0.50 × ToolCalling + 0.30 × HumanEval + 0.20 × MMLU (scale 0-100) https://github.com/raullenchai/Rapid-MLX https://github.com/raullenchai/Rapid-MLX
- JumpCrisscross 6mo agoPardon the silly question, but why do I need this tool versus running the model directly (and SSH’ing in when I’m away from home)?
- Atotalnoob 6mo agoQwen is probably your best bet… Edit: I’d also consider waiting for WWDC, they are supposed to be launching the new Mac Studio, an even if you don’t get it, you might be able to snag older models for cheaper
- hobom 6mo agoThey are taking in more than they are spending hosting them. However, the cost for training the next generation of models is not covered.
- bandrami 6mo agoNope. They're losing money on straight inference (you may be thinking of the interview where Dario described a hypothetical company that was positive margin). The only way they can make it look like they're making money on inference is by calling the ongoing reinforcement training of the currently-served model a capital rather than operational expense, which is both absurd and will absolutely not work for an IPO.
- dgellow 6mo agoDo you have sources? I would be interested to read them
- bandrami 6mo agoProbably the best roundup is Ed Zitron at https://wheresyoured.at https://wheresyoured.at Half the articles are paywalled but the free ones outline the financial situation of the SOTA providers and he has receipts
- wild_egg 6mo agoInference, in and of itself, can't be completely unprofitable. Unless you're purely talking about Anthropic? But > If you want LLMs to continue to be offered we have to get to a point where the providers are taking in more money than they are spending hosting them Suggests you just mean in general, as a category, every provider is taking a loss. That seems implausible. Every provider on OpenRouter is giving away inference at a loss? For what purpose?
- bandrami 6mo agoFor the same reason that Amazon operated at a loss for two decades and Uber operated at a loss for a decade and a half. The problem is the free money hose isn't running anymore.
- baruch 6mo agoIf they started doing caching properly and using proper sunrooms for that they'd have a better chance with that
- bandrami 6mo agoIf my empty plate had a pizza on it it would be a good lunch
- Larrikin 6mo agoIt is nobody's responsibility to ensure billion dollar companies are profitable. Use them until local models are good enough
- holoduke 6mo agoLike with all new products. It takes time to let the market do its work. See if from a positive side. The demand for more and faster and bigger hardware is finally back after 15 years of dormancy. Finally we can see 128gb default memory or 64gb videocards in 2 years from now.
- vegnus 6mo agoI'll take local models over these corporate ones any day of the week. Hopefully it's only a matter of time