6 ms·
This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you ca
by miki123211 5mo ago
This is, sadly, obvious and inevitable in retrospect.
The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, and it's much easier to move your AI workload than to move yourself.
This is a completely different dynamic than renting houses or apartments, as you can't really rent out the same house to different people at different times of day.
- cootsnuck 5mo agoYea. LLM inference requires batch processing to have a shred of hope at being cost efficient. Batch processing requires a not so insignificant amount of scale (but probably not as much as people think). I'm very pro local models, but not to have parity with SoTA frontier models. Just contextually trained small models doing smaller specific tasks. Trying to run bigger LLMs for an individual user to do big tasks is not going to be a good time.
- MichaelZuo 5mo agoWasnt this pretty evident to pretty much anyone who knew even a bit about inferencing? Idk what people were thinking. I’ve never seen anyone offer a plausible way to sidestep batch processing for example.
- miki123211 5mo agoI think people overstate the importance of batch processing and understate the importance of consistent workloads. A GPU is much like a hotel or an airplane. To make hotel and airplane economics work out, you need to keep your seats / rooms filled at all times. Because the number of spots is limited, an empty seat, room or request slot today is an opportunity forever wasted. Unlike hotels and airplanes, though, GPU requests are short-lived. This means you need somebody to provide you with useful work to do 24/7. If you're an individual who needs to sleep or a single-country business that works 9 to 5, most of your request processing opportunities will go to waste. You're much better off renting your GPU from somebody who can provide it with useful work 24/7, and hence get much better per-request prices.
- adrianN 5mo agoHistorically it was not uncommon for beds to be rented out to multiple people.
- consp 5mo agoSee military submarines, for a modern version.
- eecc 5mo agoIt is unfortunately still common practice among irregular agricultural workers in many parts of the world (I’m Italian so I definitely remember news about busts in southern Italy)
- bredren 5mo agoThe word for this type of boarding is “flophouse.” This is the type of place one might be “waiting for the other shoe to drop.” Which carries a variety of potential meanings in this moment of AI. Tangentially related: Mack and the boys lived in the “Palace Flophouse and Grill” in Cannery Row. I suppose I must have looked up flophouse when reading all the Steinbeck I could get my hands on and it’s stuck w me.
- AdamN 5mo agoYeah there are good accounts of this in Down and Out in Paris and London and also one of Hemingway's books - forgot which one.
- zozbot234 5mo agoYou can definitely run many requests in parallel as a single user, you just have to be OK with a significant slowdown for any single request. Cloud inference can't reach that ratio of total throughput per hardware cost since they are heavily incented to get the most expensive hardware available and to then minimize latency (and RAM occupation over time) even at the cost of throughput. Running slower inference with cheaper hardware is just not workable in a cloud setting.
- Unit327 5mo agoIt also doesn't help that they probably sell tokens below cost.
- PowerElectronix 5mo agoOn top of that, AI providers are also eating a big loss on the service.
- tempay 5mo agoAre they? I only ever see unsubstantiated claims for this whereas I see many justifications that interference is comfortably profitable in isolation.
- ai_fry_ur_brain 5mo agoIts basic math, go calculate max sessions for a certain tps on any hardware. Session# * tps * 86400 (secs in a day) * 30 days. You'll realize real quick its not profitible. You cant just say things you don't like to hear are unsubstantiated without verifying. Not to mention, subscriptions.. $2mm in GPUs being given out for 5 hrs a day at a cost of $200 a month. I could easily say that everyone who says its profitible is msking unsubstantiated claims lol.
- adastra22 5mo agoYou got numbers? Because it seems perfectly possible to me. OpenAI and Anthropic’s marginal cost for inference is certainly far less than their API pricing.
- ai_fry_ur_brain 5mo agoHow can you say that with such certainty? You have no idea what it costs to run a 10T parameter model at extremely high concurrency. These 1T param models running at <$3.00 per 1mm are certainly not profitable.
- adastra22 5mo agoBecause I’ve looked at what it would cost my company to self-host a SOTA sized model. For us it wasn’t worth it because the hardware is all bought up by frontier labs and we can’t get any supply. But if we could, at the prices they’re paying, it would pay for itself in 10-ish months. I assume further that they have economies of scale on top of what I was estimating.
- graemep 5mo agoHigh usage seems to change the economics. The author of the article had a payback period of about 14 months which is excellent by any standards and an order of magnitude better than rent vs buy for a house in most places.
- dpark 5mo ago> You can't get cheaper GPUs You absolutely can. OpenAI et al are paying a fortune for GPUs but they are not paying retail prices. The entire business model of retail is to sell above cost.