8 ms·
I work for a tiny little company ($150MM annual rev with 9% net) and we are already looking at dropping $100k on hardware to run local models because, for us, t
by FuriouslyAdrift 5mo ago
I work for a tiny little company ($150MM annual rev with 9% net) and we are already looking at dropping $100k on hardware to run local models because, for us, they're "good enough."
Our estimated spend for AIaaS would exceed that cost in less than a year.
In a few years, there will be hardware capable of running frontier models good enough for most things at accessible prices for even tiny companies.
- EvanAnderson 5mo ago> ...we are already looking at dropping $100k on hardware to run local models... Just think how much further that $100K would have gone if the hardware market wasn't so screwed-up. Anecdote: I priced-out adding 1TB of RAM to a four node cluster a couple months ago. The cluster was purchased in fall of 2024 w/ 4 nodes, each with 256GB RAM. The nodes cost just over $14K apiece back in 2024 (entire box, not just the RAM). Dell wanted >$90K a couple months ago to add 256GB to each node.
- cyberax 5mo ago> Dell wanted >$90K a couple months ago to add 256GB to each node. RAM is expensive, but not THAT expensive. I just bought 128Gb for about $5k for our build cluster (it's not even for AI, sigh). Even if you need larger-sized DIMM sticks, it's still going to be in the vicinity of ~15k tops.
- deleted 5mo ago[deleted]
- EvanAnderson 5mo agoIt was crazy. I found the part on the open market for a lot less but the edict from the Customer was to buy from Dell to keep the support entitlement intact. That inflated the price to an astronomical level to be sure. I haven't had problems w/ Dell support and 3rd party memory, personally, but given the machines' application I understood the concern.
- alex_suzuki 5mo agoI’m curious: are you spending on beefy developer machines, or some kind of shared local inference server? Would be interested to know more if it’s the latter.
- irishcoffee 5mo agoI am aware of at least a handful of companies doing the latter. I don’t work for them and cannot speak to their setup.
- nonethewiser 5mo agoWhat models? Last I tried different local modals there was a pretty big difference from frontier.
- simplyluke 5mo agoYeah, that's the part that just seems to be wildly under-discussed to me. If open source models are ~3-6 months behind SOTA, and ~opus4.6 capabilities are good-enough for product market fit, do the frontier labs have half a decade to catch up on their prior burn? AI cost ballooning faster than companies can afford is becoming a very common topic in my circles right now. The era of "I'll pay infinitely more for marginal gains" is over from what I can tell.
- doug_durham 5mo agoOpen source models that you can run locally are much more than 3 to 6 months behind. 6 months was the November inflection for Claude. No open source model is as good as Claude Opus 4.6.
- simplyluke 5mo ago> that you can run locally That's doing a lot of work here. The future I see isn't most companies buying hundreds of thousands in hardware to run models, it's them adding a line item to their AWS bill. Inference costs on the larger hosted open source models are dramatically lower than the frontier labs API pricing.
- apocalyptic0n3 5mo ago> it's them adding a line item to their AWS bill That's the future Amazon sees too. We just had a week long session with the AWS team and they pushed that to us multiple times.
- teiferer 5mo agoThe future I'm seeing is AI coprocessors running inference locally in most devices that today have a CPU. Just look at how powerful your mobile phone has become compared to your desktop computer 15 years ago and compared to a main frame 30 years ago. The days of requiring a data center to run anything resembling opus 4.6 are already counted. (But the industry will fight hard to get people to keep paying the Claude tax.)
- awesome_dude 5mo ago> In a few years, there will be hardware capable of running frontier models good enough for most things at accessible prices for even tiny companies. I was going to say - the models are just going to keep growing at a pace exceeding the pace of hardware pricing/availability But then I realised that, far more likely, there will be a plateau reached (again) where nobody is seeing gain, and at that point hardware will catch up
- arbuge 5mo ago> In a few years, there will be hardware capable of running frontier models good enough for most things at accessible prices for even tiny companies. What makes you so confident about this prediction? Hardware costs haven't exactly been cratering recently.
- sofixa 5mo ago.> Hardware costs haven't exactly been cratering recently. No, but local models have been booming in performance/quality improvements. The RAM shortage won't last forever (more supply will come online when if demand doesn't diminish), and then the math would be pretty easy.
- mv4 5mo agoI configured a dual DGX Spark cluster, and it's certainly "good enough" for my agentic and coding needs.
- datadrivenangel 5mo agowhat models are you using on that? My experiences with apple hardware have convinced me that it is not really good enough for coding locally.
- irishcoffee 5mo agoIt isn’t the models, it’s the closed api and the tooling associated with it. It’s driving me crazy how not-talked-about this is.
- datadrivenangel 5mo agoAs in the coding harnesses?
- irishcoffee 5mo agoIf I could leverage the same closed api VSCode uses, the entire moat is drained. What you call harnesses I call… bullshit?
- klausa 5mo agoYou can point both Codex and Claude Code at a local model and they'll work just fine. Codex even explicitly supports that as a feature! [1] With a nice UI on top, for the desktop app too: [2] [1]: https://developers.openai.com/codex/config-advanced#custom-model-providers https://developers.openai.com/codex/config-advanced#custom-m... [2]: https://docs.ollama.com/integrations/codex-app https://docs.ollama.com/integrations/codex-app
- girvo 5mo agoDeepSeek v4 Flash, various quantised versions of Kimi K2.6, MiniMax 2.7, Qwen 3.5 “full sized, with a dual spark setup you can fit some decent setups on here My single spark has me running Qwen 3.6 27B and antirez’s specially quantised DeepSeek v4 Flash (which is shockingly impressive)
- cmdrk 5mo agoDo you think this will be a trend for larger companies as well? The decadal move to all-cloud-all-the-time killed off in-house hardware teams while the C-suite chased their OpEx dreams. It would be interesting if we come full circle on this.
- fragmede 5mo agoI doubt it. Companies that have moved to the cloud are already trusting the cloud with their IP. You can rent time on a high end Nvidia system from various clouds. OpEx means there's no write down in three/five years as that system goes out of date so it would only make sense if the performance/$ is there, or the company is highly protective of their IP and doesn't trust the cloud, at which point they're not on the cloud anyway.
- MASNeo 5mo agoOn prem AI makes sense for more than just the cost. More control, IP, model improvements you can keep, data privacy to name a few. People will realize that AI is not like compute the moment they get their own knowledge sold back at a premium.
- fragmede 5mo agoWhat are the advantages to on-prem for a company that's already in the cloud and trusts it with their IP? That company can just rent GPU instances from the cloud if they want to train/fine-tune their own models and keep avoiding CapEx.
- amelius 5mo ago> People will realize that AI is not like compute the moment they get their own knowledge sold back at a premium. But what if your competitors sell their knowledge to AI companies? Then you're still screwed.
- disiplus 5mo agosame, but you need more then 100k of hw to run something like kimi k2.6 for a bigger team. on the other hand there is a ds4 flash that you can run on a macbook with 128gb ram. an that one is perfectly usable for a lot of tasks. https://github.com/antirez/ds4 https://github.com/antirez/ds4
- FuriouslyAdrift 5mo agoI think the quote came out to $107k. 4 AMD MI300A's. Around 60k tokens per second, 512GB of GPU memory. https://www.gigabyte.com/Enterprise/GPU-Server/G383-R80-AAP1 https://www.gigabyte.com/Enterprise/GPU-Server/G383-R80-AAP1
- disiplus 5mo agoWhich model are you running ?
- deleted 5mo ago[deleted]
- stopachka 5mo agoI don't quite understand, what would 100K buy you? AFAIK you would get about ~5 concurrent users, with a max context window of ~128K tokens on the larger models. This wouldn't be good enough for coding -- are you guys thinking of using it for something else?
- bendews 5mo agoBy my calculations 100k could get you 18 5090's + compute to host them, or 18 96gb Mac mini's. You can get a lot of context window and users out of that setup.
- FuriouslyAdrift 5mo agoGigabyte 4x AMD Instinct MI300A rack server (512GB GPU RAM total) Roughly equivalent to 4x H200's for less than half the price. Vaguely around 60k tokens per second...
- rstuart4133 5mo agoI get the impression the hive mind hasn't come to terms with the point that a model is optimised for certain tasks. It's like having someone ask you "is that a good hammer?". Good for what? There are claw hammers, sledgehammers, ball-peen hammers, club hammers, mallets, .... Yes, in a pinch, they can all bang in nails, but you wouldn't choose a dead blow hammer for that if you had a choice. The Gemini Flash is very good at searches. Just about any low end model can toss out a poem. All the higher end models (open source and otherwise) seem to be able to churn out code that passes tests. The smaller, "less capable" ones are much faster at it, which means in the hands of a skilled practitioner are the best choice for that task. But they rapidly fall apart where there isn't a hard source of truth (like a good test suite) to grind against. Because of that you have to use a bigger model for bug finding. In that task the open source models tend to fail on larger code bases, where something like Opus still shines. I gather Mythos is an absolute monster, and unparalleled, and unavailable. I'm sure one of the reasons for that is it's so expensive to run. Or to put it another way - you don't use a 100 tonne crane to pick up the shopping. And ... the smaller models will happily run on in-house hardware. You may not do it today because of the current DRAM price and integrated NPUs have just started shipping, but in 5 years time models will be running on your phone.
- Npovview 5mo agoYes exactly, we will have specialized models soon. These will be trained with plugin architecture with a core reasoning model asking plugin models to do stuff on its behalf. I don't need chinese or russian knowledge in my workflow.
- nkhs89 5mo ago[dead]
- nkhs89 5mo ago[dead]
- try-working 5mo agoWhat about using DeepSeek API? Practically free.
- ai_fry_ur_brain 5mo agoYou people are delusional. How many times a day am I going to read this fiction of "good enough in a few years for most things". There are physical limits to how much you can compress data and how much is needed for a capable model. If by hardware capable for running SOTA you mean a 7 figure investment for a company, than sure. But how come these companies didnt do the same thing for cloud? There's been this option for self hosting infrastructure for a decade but companies don't use it, they pay AWS.
- wyager 5mo ago> there will be hardware capable of running frontier models The current frontier? Sure. The frontier then? No - obviously that frontier is going to keep consuming available datacenter compute capacity, which will be better
- slashdave 5mo agoIt might be possible that in a few years someone will be able to engineer a reasonably priced machine to run today's frontier models (hint, your price is an order of magnitude off). However, they won't be able to run the frontier models that will exist in a few years.
- AndrewKemendo 5mo agoThat’s exactly where the market is heading and it’s going to have to reckon with this fact My guess is there’s gonna be some legislation or something “you can’t share anything over this level of complexity” and I think that that’s what a lot of that mythos rattling was all about
- safety1st 5mo agoYes 100% this. A lot of people keep talking about how OpenAI and Anthropic will need to raise their prices. What is less discussed is how they CAN'T raise their prices because competition exists, and sure it's not SOTA, but it's literally an order of magnitude cheaper in many cases and the drive to figure out how to make it work well enough is going on right now (and will only intensify when the SOTA models raise their price). It's a given that the SOTA models need to raise their prices. It's also a given that they can't. The more they raise the more customers will move to their competition. So what happens next? Well I think it will suck horribly if you can't move off of SOTA sooner or later, because the Big Two are going to lose customers, and therefore have to raise prices on the locked in customers even more than these projections suggest. Beyond that if you're looking to start a business, figure out how to use cheap models in new scenarios. Build software which does that and license it. This is kind of contrary to the idea that you shouldn't over optimize for deficiencies in the models that will likely go away in the next generation - for instance a lot of problems were solved when context windows got way bigger. So it's a thin line to walk but I think it's there because a lot of orgs are using Claude today for pretty basic tasks. The dev who's addicted to SOTA models honestly is going to have to settle for less or get totally screwed. Most applications within business from what I see aside from complex research do not require SOTA. They summarize, they classify, they transform, and doing that accurately has been cheap for a while.
- extraextra 5mo agoI'd qualify your point that Anthropic and OpenAI can't raise prices, that is as of right now. Once the industry will go through a phase of consolidation and the bigger players will have some moat around their product, they'll have more pricing power. Your last point is common sense in my opinion, I agree with it. At the end of the day most employees are (by definition) of average intelligence and most businesses are average in complexity. Thus, it is logical that average tools (AI models) should do the job for most people and most businesses.
- fittingopposite 5mo agoAgree. You have these tipping points when a model is good enough to do some task. Yes, a better model will further improve your capabilities but the unlock is at a certain intelligence level. We see this also with humans. People with very low intelligence can't learn to read. Once you cross a certain threshold of intelligence you can learn to read. More intelligence doesn't really help you in the task of reading. A person with an IQ of 160 is not substantially better in reading than someone with an IQ of 85. If your IQ is 50, you might not be able to learn to read at all.
- teaearlgraycold 5mo agoHave you considered that a smarter person will understand what they have read better?
- physicsguy 5mo agoDepends on the task and the writing though doesn't it? There's not that much depth in a lot of 'everyday' writing. For many tasks that means that you don't need to be hyperintelligent - reading a recipe or a shopping list, reading a newspaper article, etc.
- physicsguy 5mo agoMy much larger company has got people already using various models through Bedrock because the Claude and OpenAI limits are too harsh and it's too expensive.
- amelius 5mo agoEh, one question. Where do you intend to buy the hardware if datacenters take over the market?
- soonnow 5mo agoI see even smaller companies ask for that. Less about cost, European companies have a deep mistrust of US companies at the moment. I see companies a tenth of your companies size ask about local models.