8 ms·
> They will be, and that moment is not that far off. It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (th
by reisse 5mo ago
> They will be, and that moment is not that far off.
It's here, right now. I'm running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it's slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and even write code when little context is required. Probably I could get a half-decent autocomplete out of it, if I bother with VS Code integration. "128 GB VRAM on a MacBook Pro or a Strix Halo" is already a minimum viable setup for agentic coding, I think.
> And then we'll have the equilibrium we already have with the "classic cloud": you either self-host or pay for flexibility and speed.
Currently, it works exactly the other way. The cloud versions are orders of magnitude cheaper than self hosting, because sharing can utilize servers much more efficiently. Company can spend half a million bucks on a rig running GLM 5.1, and get data security, flexibility and lack of censorship, but oh it's so expensive compared to Anthropic per-seat plans.
- datadrivenangel 5mo agoIn my experience once you get to ~30 gigs of ram for a model like Gemma4, the rest of the 128g of memory is simply nice to have. The speed and costs are what make it tough though, because its slower and more expensive than the same model served on a big accelerator card, and is going to be worse than a frontier model.
- digitaltrees 5mo agoI wonder if it really needs to be worse. I am playing with the idea of fine tuning a model on my exact stack and coding patterns. I suspect I could get better performance by training “taste” into a model rather than breadth.
- epicureanideal 5mo agoI also wonder about JS only, Python only, etc models. Maybe the future is a selection of local, specific stack trained models?
- andy_ppp 5mo agoThese models being able to generalise at coding will likely get worse if you remove high quality training data like all of python.
- robrenaud 5mo agoThere is some recent work on modularizing knowledge in LLMs. https://arxiv.org/html/2605.06663v1 https://arxiv.org/html/2605.06663v1 It might be possible to train a big generalist that is a composition of modules, some of which can be dropped dynamically at inference time, depending on the prompt.
- digitaltrees 5mo agoCool. Thanks for sharing. I am thinking about creating a series of smaller models for specific purposes and then orchestrating them so they mirror the human brain which is a bunch of subsystems that give multiple opinions about the same stimulus
- shailendra_sis 5mo agoInteresting direction. I’ve also been thinking about modular / subsystem-based approaches for specialized tasks in small AI systems.
- andy_ppp 5mo agoFine tuning these models (at least with PPO or equivalent) requires even more VRAM than inference does, potentially 2-3 times more.
- jimbokun 5mo agoThat approach has its advantages, but sometimes I want to generate code for a language or kind of project I’m not experienced with using the accepted best practices.
- ElectricalUnion 5mo agoYou need the rest of the ram for the context. If you don't want to end up with a toy context or quantized lossy context, is pretty easy to end up having to spend up 50+GB just for the KV cache, per simutaneous inference slot.
- zozbot234 5mo ago[dead]
- winocm 5mo agoPerhaps I am the odd one out here, but a small part of me wants to see what happens when you run a proprietary SOTA model on a laptop.
- amelius 5mo agoYou burn your lap?
- reisse 5mo agoNothing special? I mean, inference engine might need to get some tweaks, to support whatever compute is available. But then, if you put a few terabytes of disk for swap, and replace RAM to bigger sticks if possible, it should work? Slowly, of course, but there is no reason it should not to.
- reverius42 5mo agoThe big difference will be measuring seconds per token instead of tokens per second.
- martijnvds 5mo agoSeconds per token is just fractional tokens per second ;)
- degamad 5mo ago> fractional Reciprocal?
- yfw 5mo agoYou can if you have enough ram slots?
- SilentM68 5mo agoNot sure if this is exactly the scenario you envision but I run ComfyUI on an Acer Helio 300 laptop, from four years ago. Has 16GB RAM, NVIDIA GeForce RTX 2060 w/6144MiB of VRAM and have generated a few images using "NetaYumev35_pretrained_all_in_one.safetensors" @ 10.6GB checkpoint, (well beyond the 6GB capacity of the RTX 2060 card). That being said, it takes more than 10 minutes to complete the task. Of course, I have to turn off all other apps, and browser tabs or hibernate them. If I don't, the laptop's fans begin to spin up like an airplane propeller. It's worth mentioning that I've tried to do this with other IDEs and all seem to fail with some error or another, usually out of VRAM issue. I've only gotten it to work with ComfyUI. I use an anaconda environment, though would have preferred an "uv" environment, on Linux and automate the startup sequence using the following script (start_comfy.sh) from the term rather than manually starting the environment from same said term: #!/bin/bash # # temporary shell version eval "$(conda shell.bash hook)" conda activate comfy-env comfy launch -- --lowvram --cpu-vae Here are some of the images: https://imgbox.com/nqjYhdx3 https://imgbox.com/nqjYhdx3 https://imgbox.com/93vSWFic https://imgbox.com/93vSWFic https://imgbox.com/qs1898dz https://imgbox.com/qs1898dz I'm hesitant to increase the sizes of the renders as that will surely stress my laptop's components.
- digitaltrees 5mo agoI built my own IDE and run my own model specifically to have private agentic coding. I can still access model APIs but I can be purely local if I want too. It’s amazing.
- manmal 5mo agoCurious, why did Zed with ACP not work for you?
- Fokamul 5mo agoI'm just guessing, but IDE which is using 3D acceleration just for stupid UI to run "smoothly", that is ridiculous. Who runs IDE with LLM agents accessing your local filesystem, on bare metal? Or am I alone to run everything LLM related on my VM just for development work. Then because of ZED genius decision, you need to share your GPU to VM, then some important features will not work, like snapshots. So you also need workaround for this, etc. Too much hassle, Zed is not for me. But I'm anti-Apple, so maybe that's the reason :) Btw, even "ImHex" devs realized this and they're providing version without acceleration for VM use. They're using ImGui. Using it for local desktop app UI is also ridiculous, imho. Whatever.
- oslem 5mo agoI would imagine running a local LLM for development isn’t as popular as using a hosted provider. I don’t personally host a local model, but I have shared GPUs and storage volumes with VMs and I didn’t see it as that much of a hassle. What kinds of problems are you running into? Doesn’t ghostty also use graphics acceleration? I was under the impression that rendering text is a relatively challenging graphics compute task.
- digitaltrees 5mo agoI run local LLM on my MacBook together with frontier models for different tasks. I am in the process of setting up a 3 Mac studio system to serve AI to my team.
- antidamage 5mo agoThis is my exact setup as well and dear lord gemma is absolutely batshit insane. I'm trying to get a self-reflection and confidence loop going now, but it does feel like it's not the local resources, it's the limits of the training. Dedicated coding or dedicated real-world task models would be a good optimisation.
- yieldcrv 5mo agoI need to see these proper harnesses I tried oMLX and OpenCode a few weeks ago and the 65k context window was useless, it tried to analyze a very small codebase before going full on agentic and ran out of context window immediately I don't have time to tweak 1,000 permutations of settings just re-prove that its not as smart as Opus 4.6 I need out the box multimodal behavior as similar as typing claude in the command line and its so not there yet but I'm open to seeing what people's workflows are
- nullsanity 5mo agoHey man, you can just say "I'm lazy, so I'm staying with the cloud. if I wanted to use my brain, I wouldn't be using AI, gosh" - it's much shorter.
- fennecfoxy 5mo agoPersonal attacks are against the rules, by the way.
- yieldcrv 5mo agoall the money and clout is in considering people’s reported problems as valid and solving them so when I encounter a common but invalidated friction, I explain it like I’m 5, understanding that many of the engineering and entrepreneurial problem solvers have the emotional intelligence of a 5 year old
- phamilton 5mo agoI'm running opencode with qwen3.6-35b-a3b at a 3-bit quant. I also have qwen3.5-0.8b used for context compaction. I run with 128k context. It's usable. I set it loose on the postgres codebase, told it to find or build a performance benchmark for the bloom filter index and then identify a performance improvement. It took a long time (overnight), but eventually presented an alternate hashing algorithm with experimental data on false positive rate, insertion speed and lookup speed. There wasn't a clear winner, but it was a reasonable find with rigorous data.
- 5mo ago
- DrewADesign 5mo agoMultiple gazillion dollar companies each seem to be spending to ensure that they alone pretty much dominate all knowledge work, with customers eating up their tokens like Cookie Monster. I wonder if the any of them could survive as LLM providers if they not only failed to do that, but the entire industry ended up selling what the current Cookie Monster would call a “sometimes snack,” for very special occasions?
- dust1n 5mo agoCan you share how you use it to categorize trip photos!
- Mario9382 5mo agoI'm also interested on how to do this
- Farmadupe 5mo agoI'm not sure there's a one-stop shop for this at the moment. I think the process is: * Have a box with sufficient spare (V)RAM -- probably 8G for simple categorization with qwen3.5-4b, and 24G or more for more intelligent categorization with qwen3.6-27b or gemma4-31b. * Download or compile llama.cpp. Choose a model, then choose one of the "quantized" builds that will actually fit on your hardware. There are literally hundreds to thousands of these per model on Hugging Face. * Spend half a day tuning command-line parameters until llama.cpp doesn't crash. * Watch llama.cpp regularly OOM itself, then put it in a systemd service with a memory limit so it doesn't take the entire machine down when it dies. * Download all your photos to a folder. * Start vibing a Python script to categorize your images by repeatedly prompting the LLM with each image in turn. * Spend days tweaking/refining the prompt to try to get the LLM to actually do what you want. The endgame is one of: * The local model categorizes your images. Yay. * The local model is too slow and you give up. Boo. * The local model is too slow, so you spend $1k-$10k on hardware. Your image categorization task becomes a cover story for buying new gear. Yay. * The local model can't understand your categorization metric, so you give up. Boo. * You eagerly await news of the next open model being released. Yay? * You consider replacing your local model with a frontier model, but then you realize you'd be spending $500 to categorize your photos. Boo. * You refuse to allow Google/Gemini/Anthropic to train on your nudes. Boo.
- creativeSlumber 5mo agothis is one of the most popular options. Self hosted. https://immich.app/ https://immich.app/
- pbgcp2026 5mo agoI'm sorry to spoil it for you, but Perl script was able to do all of that like ... 10 years ago? The out-of-the-box Shotwell manages photos quite well without any intelligence. The problem, as people mentioned above, is SOTA models cognitive and tooling abilities. Also, have you noticed as top-end Mac Studios got downgraded recently? They don't want you to have access to frontier models. And you will not have it. See Mythos as Exibit A.
- Hamuko 5mo ago>Also, have you noticed as top-end Mac Studios got downgraded recently? They don't want you to have access to frontier models. And you will not have it. Isn't that a function of RAM supply not being available now?
- aceazzameen 5mo agoOpenAI did buy out the RAM supply to block competition. Arguably local models are one of its (smaller) competitors. Even if that weren't the case, every corp _needs_ you to be on a subscription.
- Hamuko 5mo agoThey didn't really even buy the RAM. But there's pretty significant demand for RAM in general with data centers being planned left and right.
- ubercore 5mo agoThe conspiracy angle here is not really relevant. Ram is expensive and they're gearing up for M5 studios. Not the illuminati keeping better LLM models out of your hands.
- lkjdsklf 5mo agoThey did decrease the memory bandwidth for.... reasons... which didn't make much sense.. but yeah this is some pretty weird conspiracy stuff. Apple doesn't even sell a model. They just have a deal to use Googles. They can't "protect" their cloud version of a model they don't have.
- fennecfoxy 5mo ago>It's here, right now. I mean I've been forcing my good old 1080ti to run local models since a short while after llama was first leaked. But I wouldn't say "local models are here" in the same way as "year of the Linux desktop!111" Until someone can just go out and buy some sort of "AI pod" that they can take home, plug in and hit one button on a mobile app to select a model (or even just hide models behind various personas) then I wouldn't say it's quite there yet. It's important that the average consumer can do it, I think the limitations for that are: things are changing too quickly, ram+compute components are exceedingly expensive now, we're still waiting on better controls/harnesses for this stuff to stop consumers not just from shooting themselves in the foot, but blowing their foot clean off. Would be interesting to see a Taalas-like chip in a product, albeit there's so many changes going on atm with diffusion based models, Google's Turboquant (which as someone who has had to almost always run quantized models, makes a lot of sense to me).
- skillina 5mo agoWhat is the use case you see for non-technical users self-hosting? I think it’s important that tools remain available but I don’t expect it to be adopted by “average consumers.” I’m interested in self-hosting for privacy and control. I already owned the hardware I’m testing with, so my spend is limited to time and electricity. The “LLM pods” you describe will be loaded with spyware and adware (see: Smart TVs), and average consumers won’t max their compute around the clock so naturally data centers are able to make more efficient use of hardware by maximizing utilization.
- fennecfoxy 5mo agoAgree with your point on them being loaded up with spyware etc because that's just how it is now I suppose. In terms of maximising compute I kind of agree but also kinda not - people's laptops and phones aren't burning at 100% 24/7 either. Sure AI requires so much more compute...but not _that_ much more, especially as technology marches on. For the general use case; I could be wrong but I'd see it sort of like a GPU/NAS/etc. "Pay once" rather than a subscription (to a service offered by a datacenter). But tbf, the way things are now _is_ all subscription models and consumers just kinda let it happen. I would love to be able to pay a one-off fee for lightroom...but I can't because they want a subscription to "pay for all the updating we're doing". They barely update shit.
- sanderjd 5mo agoAre there any harnesses that are attempting to optimize for using local models like this? Unsurprisingly, my naive attempts to integrate with harnesses designed for frontier models have gone poorly. But it seems like a harness that understands the capabilities and limitations better could perform significantly better.
- nsvd2 5mo agoI run Gemma locally on a 3090, it's amazing how useful it is to be able to call out to ollama in a bash script or cron job.
- jimbokun 5mo agoHas anyone tried to calculate the break even cost of buying a PC to run an LLM locally, vs the amount of tokens you could get from an AI provider?
- zozbot234 5mo agoThe basic answer: very much not worth it at face value, becomes arguably worth it once you start worrying about future rug pulls from the big AI providers. (And that does include the market for third-party inference, at least at present.) It's also worth it if you have existing hardware to repurpose, but that's obvious and not what you were asking about.
- thot_experiment 5mo agoAlso you can feed it ALL of your data willy nilly without ever worrying about safety because you can just do it with the LAN cable unplugged, for applications that demand data hygiene it's a cheat code that guarantees safety without any sort of data sanitization.