6 ms·
The models I can run locally aren't as good yet, and are way more expensive to operate. Once it becomes economical to run a Claude 4 class model locally you'll
by simonw 1y ago
The models I can run locally aren't as good yet, and are way more expensive to operate.
Once it becomes economical to run a Claude 4 class model locally you'll see a lot more people doing that.
The closest you can get right now might be Kimi K2 on a pair of 512GB Mac Studios, at a cost of about $20,000.
- theshrike79 1y agoThis is the thing. I'm waiting for the equivalent of Google Coral[0], but powerful enough for AI workloads. You can plug in the $60 Coral to a Raspberry Pi and get real-time image recognition running in Frigate. When I can have: 1) Something similar inside my computer/laptop 2) Something I can plug in to my computer via USB-C 3) Something I can buy and install to my LAN so all devices in my home can connect to it I'll buy it instantly. What I don't want is a massive generic GPU that just happens to be good at AI workloads, I want custom hardware that's more efficient and cheaper. (Off topic, but my guess is that Apple is aiming for #3 with an Apple TV variant so you can have more power than your phone, but still keep it 100% local) [0] https://coral.ai/products/accelerator/ https://coral.ai/products/accelerator/
- deleted 1y ago[deleted]
- QRY 1y agoHave you considered the Framework Desktop setup they mentioned in their announcement blog post[0]? Just marketing fluff, or is there any merit to it? > The top-end Ryzen AI Max+ 395 configuration with 128GB of memory starts at just $1999 USD. This is excellent for gaming, but it is a truly wild value proposition for AI workloads. Local AI inference has been heavily restricted to date by the limited memory capacity and high prices of consumer and workstation graphics cards. With Framework Desktop, you can run giant, capable models like Llama 3.3 70B Q6 at real-time conversational speed right on your desk. With USB4 and 5Gbit Ethernet networking, you can connect multiple systems or Mainboards to run even larger models like the full DeepSeek R1 671B. I'm futsing around with setups, but adding up the specs would give 384GB of VRAM and 512GB total memory, at a cost of about $10,000-$12,000. This is all highly dubious napkin math, and I hope to see more experimentation in this space. There's of course the moving target of cloud costs and performance, so analysing break-even time is even more precarious. So if this sort of setup would work, its cost-effectiveness is a mystery to me. [0] https://frame.work/be/en/blog/introducing-the-framework-desktop https://frame.work/be/en/blog/introducing-the-framework-desk...
- cheeze 1y agoI love Framework but it's still not enough IMO. My time is the most valuable thing, and a subscription to $paid_llm_of_choice is _cheap_ relative to my time spent working. In my experience, something Llama 3.3 works really well for smaller tasks. For "I'm lazy and want to provide minimal prompting for you to build a tool similar to what is in this software package already", paid LLMs are king. If anything, I think the best approach for free LLMs would be to run using rented GPU capacity. I feel bad knowing that I have a 4070ti super that sits idle for 95% of the time. I'd rather share an a1000 with bunch of folks and have that run at close to max utilization.
- generic92034 1y ago> and a subscription to $paid_llm_of_choice is _cheap_ relative to my time spent working. In the mid to long term the question is, is the subscription covering the costs of the LLM provider. Current costs might not be stable for long.
- RugnirViking 1y agoThat is, in every sense of the term, their problem. I will switch to whatever is best for me at a good price, and if thats not sustainable then I'll be fine too; I was a developer before these existed at all, and local models only help from there.
- jakebennet89 1y ago[dead]
- lhl 1y agoStrix Halo does not run a 70B Q6 dense model at real-time conversational speed - it has a real-world MBW of about 210 GB/s. A 40GB Q4 will clock just over 5 tok/s. A Q6 would be slower. It will run some big MoEs at a decent speed (eg, Llama 4 Scout 109B-A17B Q4 at almost 20 tok/s). The other issue is its prefill - only about 200 tok/s due to having only very under-optimized RDNA3 GEMMs. From my testing, you usually have to trade off pp for tg. If you are willing to spend $10K for hardware, I'd say you are much better off w/ EPYC and 12-24 channels of DDR5, and a couple fast GPUS for shared experts and TFLOPS. But, unless you are doing all-night batch processing, that $10K is probably better spent on paying per token or even renting GPUs (especially when you take into account power). Of course, there may be other reasons you'd want to inference locally (privacy, etc).
- zer00eyz 1y ago> Once it becomes economical to run a Claude 4 class model locally you'll see a lot more people doing that. Historically these sorts of things happened because of Moores law. Moores law is dead. For a while we have scaled on the back of "more cores", and process shrink. It looks like we hit the wall again. We seem to be near the limit of scaling (physics) we're not seeing a lot in clock (some but not enough), and IPC is flat. We are also having power (density) and cooling (air wont cut it any more) issues. The requirements to run something like claud 4 local aren't going to make it to house hold consumers any time soon. Simply put the very top end of consumer PC's looks like 10 year old server hardware, and very few people are running that because there isn't a need. The only way we're going to see better models locally is if there is work (research, engineering) put into it. To be blunt that isnt really happening, because Fb/MS/Google are scaling in the only way they know how. Throw money at it to capture and dominate the market, lock out the innovators from your API and then milk the consumer however you can. Smaller, and local is antithetical to this business model. Hoping for the innovation that gives you a moat, that makes you the next IBM isnt the best way to run a business. Based on how often Google cancels projects, based on how often the things Zuck swear are "next" face plant (metaverse) one should not have a lot of hope about AI>
- esafak 1y agoModel efficiency is outpacing Moore's law. That's what DeepSeek V3 was about. It's just we're simultaneously finding ways to use increase model capacity, and that's growing even faster...
- zer00eyz 1y ago> Model efficiency is outpacing Moore's law. Moores law is dead, has been for along time. There is nothing to outpace. > That's what DeepSeek V3 was about. This would be a foundational shift! What problem in complexity theory was solved that the rest of computing missed out on? Don't get me wrong MOE is very interesting but breaking up one large model into independent chunks isn't a foundational breakthrough its basic architecture. It's 1960's time sharing on unix basics. It's decomposition of your application basics. All that having been said, there is a ton of room for these sorts of basic, blood and guts engineering ideas to make systems more "portable" and "usable". But a shift in thinking to small, targeted and focused will have to happen. Thats antithetical to everything in one basket throw more compute at it and magically we will get to AGI. That clearly isnt the direction the industry is going... it wont give any one a moat, or market dominance.
- smallerize 1y agoI don't have to actually run it locally to remove lock-in. Several cloud providers offer full DeepSeek R1 or Kimi K2 for $2-3/million output tokens.
- ketzo 1y agoIn what ways is that better for you than using eg Claude? Aren’t you then just “locked in” to having a cloud provider which offers those models cheaply?
- viraptor 1y agoAny provider can run Kimi (including yourself if you would get enough use out of it), but only one can run Claude.
- hhh 1y agoTwo can run Claude, AWS and Anthropic. Claude rollout on AWS is pretty good, but they do some weird stuff in estimating your quota usage thru your max_tokens parameter. I trust AWS, but we also pay big bucks to them and have a reason to trust them.
- viraptor 1y agoIn a way... But it's still just because Anthropic lets them. Things can change at any point.
- oblio 1y agoThe thing is, code is quite compact. Why do LLMs need to train on content bigger than the size of the textual internet to be effective? Total newb here.
- airspresso 1y agoMany reasons, one being that LLMs are essentially compressing the training data to unbelievably small data volumes (the weights). When doing so, they can only afford to keep the general principles and semantic meaning of the training data. Bigger models can memorize more than smaller ones of course, but are still heavily storage limited. Through this process they become really good at semantic understanding of code and language in general. It takes a certain scale of training data to achieve that.
- oblio 1y agoYeah, I just asked Gemini and apparently some older estimates put a relatively filtered dataset of Github source code at around 21TB in 2018, and some more recent estimates could put it in the low hundreds of TB. Considering as you said, that LLMs are doing a form of compression, and assuming generously that you add extra compression on top, yeah, now I understand a bit more. Even if you focus on non-similar code to get the most coverage, I wouldn't be shocked if a modern, representative source code training data from Github weighed 1TB, which obviously is a lot more than consumer grade hardware can bear. I guess we need to ramp up RAM production a bunch more :-( Speaking of which, what's the next bottle neck except for storing the damned things? Training needs a ton of resources but that part can be pooled, even for OSS models, it "just" need to be done "once", and then the entire community can use the data set. So I guess inference is the scaling cost, what's the most used resource there? Data bandwidth for RAM?
- airspresso 1y agoYes, for inference the main bottleneck is GPU VRAM and the bandwidth between the GPU cores and VRAM. Ideally you want enough GPU VRAM to be able to load the entire model into VRAM + have room for caching the already-produced output in VRAM when you're generating output tokens. And fast enough VRAM bandwidth that you can copy the weights from VRAM to GPU compute cores as fast as possible to do the calculations for each token. This determines the tokens/sec speed you get for the output. So yes, more and faster VRAM is essential.
- NiloCK 1y ago> Once it becomes economical to run a Claude 4 class model locally you'll see a lot more people doing that. By that time Claude 5 (or whatever) will be available over API. I am grateful for upward pressure from models with published binaries - I do believe this is fundamental floor-raising technology. Choosing frontier-1 for the sake of privacy, autonomy, etc will always be a hard sell and only ever to a pretty niche market. Even me - I'm ideologically part of this market, but I'm already priced out hardware wise.
- mythz 1y agoWhilst it's not economically feasible to self-host, using premier OSS models like Kimi K2 / DeepSeek via OpenRouter gets you a great price with a fallback safety net of being able to self-host should the proprietary model Co's collude and try and squeeze more ROI out of us. Hopefully by then the hardware to run the OSS models will be a lot cheaper to run.
- jmb99 1y agoWhat’s your budget and speed requirement? A quad-CPU Xeon E7 v4 server (Supermicro X10QBI, for example) with 1TB of RAM gives you ~340GB/s memory bandwidth and enough actual memory to host a full DeepSeek instance, but it will be relatively slow (a few tokens/s max in my experience). Up front cost a bit under $1k, less if you can source cheap 32GB DDR3 RAM. Power consumption is relatively high, ~1kW under load. But I don’t think you can self host a large model cheaper than that. (If you need even more memory you could equip one of those servers with 6TB of DDR3 but you’ll lose a bit of bandwidth if you go over 2TB. DDR4 is also a slightly faster option but you’re spending 4x as much for the same capacity.)
- theshrike79 1y agoI think we're at early 2000's bitcoin markets here. People were buying stores empty of GPUs to mine for BTC. Then people built custom ASICs that couldn't do anything but mine BTC, but did it a lot cheaper and with a lot less electricity required -> nobody GPU mines anymore pretty much. I'm waiting for a similar thing to happen to local AI.
- pmarreck 1y agoThis would require massively more power than the Mac Studios.
- jmb99 1y agoYep, ~1kW as mentioned. Depending on your electrical rate, break even might be years down the line. And obviously the Mac Studios would perform substantially better. Edit: And also, to get even half as much memory, you need to spend $10k. If you want to host actually-large LLMs (not quantized/distilled versions), you'll need to spend close to that much. Maybe you can get away with 256GB for now, but that won't even host full Deepseek now (and I don't know if 512GB either, with OS/etc overhead, and a large context window).
- otabdeveloper4 1y agoa) Rent a GPU server. b) Learn to finetune your models. You're a programmer, right? Whatever happened to knowing your tools? OP is right, these people are posers and fakers, not programmers.
- simonw 1y agoHave you had any success finetuning models? What did you do?
- otabdeveloper4 1y agoNot yet. That day will come though.
- Abishek_Muthian 1y agoWhat type of code you write for which the opensource models aren't good enough? I use Qwen2.5 coder for auto complete and occasional chat. I don't want AI to edit my code and so this works well for me. I agree that the hardware investment for local AI is steep but IMO the local models are good enough for most experienced coders who just want a better autocomplete than the one provided by the IDE by default.
- arvinsim 1y agoThe people who are subscribing to private LLM models are not doing it for better autocomplete. These are the people who want more features like agents.
- InterviewFrog 1y agoUsing AI for autocomplete is like using a racecar to pick up groceries. This is exactly what the author says about avoiding LLMs for some ideological or psychological refusal.
- manmal 1y agoNothing's wrong with using autocomplete in addition to agents.
- poulpy123 1y agoYeah. I cannot even run significantly worse models on any machine I have at home.