4 ms·
Still needs 32-64GB memory to run it locally. 64GB Macbook pro with an M5 chip costs more than 4k Euros in Germany. A more practical model would be a language s
by sajithdilshan 2mo ago
Still needs 32-64GB memory to run it locally. 64GB Macbook pro with an M5 chip costs more than 4k Euros in Germany. A more practical model would be a language specific (e.g Python or JVM language) and excellent at tool calling and reasoning. Maybe that way they can shrink it even more.
- Gecko4072 2mo agoThere have been discussions on language specific not really being a relevant change to reduce size.
- Manfrednotfunny 2mo agoI would love to see any good research projects about it but i have the feeling that Frontier with MoE is making too fast of a progress so that a customized model would always be worse and that the MoE part is actually going somehow in this direction. On the other hand, at the GTC was a talk about coding in different lanugage (like spanish) and explaining that the quality between spanish and english is relevant different. But i have not found a good article about the impact of learning data with practical experiments or even if the order of the learning data matters. At least I think i remember that Meta mentioned having better and less data can be better than more data with lower quality. As long as these models can explain to you facts about any other topics, its still overfitted for the task though.
- mapontosevenths 2mo agoCapability in LLM's is distributed throughout the manifold in subspaces. Even worse, the subspaces exist in superposition. That is to say, there is no single 'python' part of the model. The python bit is spread throughout the entire model and overlaps with other pieces that have similar, but unrelated, capabilities. For example the python subpspace might be partially in superposition with cupcake recipes, Esperanto, and calculus. We need calculus in a coding agent but not the other two. However, separating them cleanly is almost impossible, and even identifying them is tough. Internally the manifolds are highly inefficient and nothing like you would imagine something humans built would be designed. It's more like something that evolved in nature.
- Manfrednotfunny 2mo agoMy current image from a MoE is that the base/core might be the more generic thing and that things like python are part of one expert though.
- mapontosevenths 2mo agoWith MOE you train a router designed to select which parts to activate. The router itself is a trained neural network and the 'experts' are usually not really things like 'python'. They're just the functional subspaces I described above. Again, those subspaces are all somehow inextricably correlated and live in complex superposition spread throughout the manifold. The router doesn't know (or care) WHY those sections get lit up it just learns which ones to activate to optimize it's own reward function. So maybe it learns to activate "logic", "python" and "cupcake recipes in esperanto" whenever it see's something that kind of looks like python. It's not the best answer, it's just the best answer the tiny router could figure out. It's all wildly complicated and inefficient, and works nothing like any reasonable human would imagine that it SHOULD operate.
- dist-epoch 2mo agoThere was some paper about routing at training bio-knowledge into a particular region of the model, which you then can cutoff when serving. But you probably lose some efficiency since maybe you sized that region too small/too big.
- mapontosevenths 2mo agoThat's a very clever approach. Any idea about the papers title or authors? I'd love to look it up.
- dist-epoch 2mo agohttps://arxiv.org/abs/2410.04332 https://arxiv.org/abs/2410.04332 https://alignment.anthropic.com/2025/selective-gradient-masking/ https://alignment.anthropic.com/2025/selective-gradient-mask...
- solarkraft 2mo agoI feel like we’ve had this discussion before. From what I remember, specialized models rarely do that much better than general ones, hence no mode Codex models.
- sparkling 2mo agoEven if you had a 64GB machine: Are you willing to reserve 90% of your memory to run a LLM? With dirt cheap models like deepseek-v4-flash that will run "forever" on $10, the answer for me is clearly: no.
- zoobab 2mo ago"With dirt cheap models like deepseek-v4-flash that will run "forever" on $10, the answer for me is clearly: no." When it's free, you are the product.
- IMTDb 2mo agoDeepseek flash is open weight, this means we can download and run that model without any connection to deepseek, no data/tokens/usage data ever reaches them. They cannot make us their product.
- prplxd_nihilist 2mo agoI see many people saying deepseek and other chinese providers have always been profitable. Also they show their training costs publicly. Can't say for sure since I have not used it personally, but I think they'll for sure outlive the western SOTAs.
- LogicFailsMe 2mo agoOpenAI apparently runs a profitable inference business with 40% gross margin, but their advertising budget is nutso and their real costs are pretraining and research. I suspect Deepseek's comp is not predicated on capturing the lightcone of all future value, some googling insinuates their top pay is $212K US which would support that suspicion. Compare and contrast with the $1.35M and up at OpenAI.
- mihaelm 2mo agoI'm sooo happy I pulled the trigger on upgrading and getting a new laptop (with 64 GB RAM) last summer. Feels like it was just in time before the exponential price jumps.
- ishtanbul 2mo agoPulled the trigger?
- idiotsecant 2mo agoCommon phrase.
- karolist 2mo agoParent used "pulled the plug", are you saying it's applicable here and not "pulled the trigger" like suggested?
- deleted 2mo ago[deleted]
- Hinrik 2mo agoThat commenter you're replying to knows that. The original commenter before them wrote "pulled the plug" which is different and doesn't quite apply here (actually implies the opposite of what they meant to say).
- idiotsecant 2mo agoAh, the edit was fast enough not to leave a mark, leaving a conversation fragment that made me think OP was being weird about a gun related phrase or something
- mihaelm 2mo agolol, you're right, the brainfart completely changes the meaning. I corrected it.
- 2mo ago
- karimf 2mo agoPractically ~20GB with KV cache > We quantize weights to ~4-bit, bringing the LM under 20 GB. We validated minimal to no degradation on agentic tasks under compression. https://www.reddit.com/r/LocalLLaMA/comments/1vkgsum/introducing_muse_glimmer_an_openweight_model/ https://www.reddit.com/r/LocalLLaMA/comments/1vkgsum/introdu...
- dbbk 2mo agoWell if you're spending thousands on API tokens already, you could just drop the same amount on a 128GB MacBook Pro and that's a one time cost.
- smallerize 2mo agoIf you're dropping thousands on API tokens, you're going to be slowed down at least 10x trying to do everything on a single MBP.
- dannyw 2mo agoBut you could grab a 5090, and paired with some DRAM for MoE offloading of bigger models, and be a happy camper with 1.8TB/s of memory bandwidth. Or just use Luna honestly. Worth considering if you’re ok with hosted APIs.
- neuroticnews25 2mo agoDon't forget about energy usage, you'll probably never break even vs same model on openrouter.
- jurgenburgen 2mo agoIf you can’t do it cheaper on your own hardware it does make you wonder how much of the cost of inference those large LLM providers are eating? Datacenter hardware isn’t magic.
- petu 2mo agoDatacenter hardware can batch at large scale, probably over 90% more energy efficient per token than a MacBook.
- flaunf221 2mo agoYour personal hardware probably isn't running useful tasks 24/7. If you spend 60% of your 8h work day on full on agentic work, then your hardware is paying off for itself only 20% of available time.
- formerly_proven 2mo ago4K bucks buys you around 180 months of <insert AI subscription here> with zero upfront cost.
- Mistletoe 2mo agoHaha wow. I’m trying to even imagine the AI landscape in 15 years and I can’t.
- butlike 2mo agoInstead of saying "I have a MBP with 64gb of RAM" you'll hear people say: "I'm subscribed to Model 9.x11B" and others will comment: "Oh dang, that's a nice model!"
- zamalek 2mo agoProblem is that might go away or get nerfed.
- ody4242 2mo agothen you switch provider, it's not a monopoly
- formerly_proven 2mo agoIf that happens you can still buy hardware later with almost certainly more (tok/s)/$ and better capabilities to run newer models more efficiently (remember native MXFP4?). Right now basically every generation of accelerator is adding new capabilities. These aren't yearly DirectX 9.0c-compatible GPU performance bumps. As an individual, for average privacy needs (e.g. open source or at-home coding and automation), it's pretty much complete nonsense financially to self-host LLMs currently or select hardware now based on the capability to do so, and pay thousands of bucks extra.
- skohan 2mo agoIf you don't mind exfiltrating all your IP to the API provider
- cynicalsecurity 2mo agoI don't understand the desire to run own AI models for programming locally. No laptop is ever going to be as powerful and energy efficient to run anything close to OpenAI, Anthropic or Google models. A model you can run on a loptop is simply not going to work as well as it's needed for programming. Small models for linguistic work fine, but anything more sophisticated simply won't provide enough resources or power. Or models would need to be significantly dumbed down - then why use them at all? So far the idea of carrying a "thin" or "thin"-like device looks more reasonable to me, while running AI on your own server.
- flaburgan 2mo agoYet.
- OtherShrezzing 2mo ago> A model you can run on a loptop is simply not going to work as well as it's needed for programming The models you can run on a high-spec laptop today are approximately where frontier models were 12-18mo ago (albeit at a lower tok/s rate). If you scan back through hn comments from that era, you’ll find plenty of people saying “this is powerful enough to massively increase my productivity”.
- anon373839 2mo ago> albeit at a lower tok/s rate Not always! I get 80-100 tok/s from Qwen 3.6 35B-A3B on a MacBook Pro thanks to MTP. With long contexts that dips to around 50-60. However, prefill is much slower than API models. So it becomes really, really, really critical to not have cache misses.
- drivebyhooting 2mo agoSlower is meaningfully dumber when you’re time bounded and need all the inference time compute you can get.
- ComputerPerson 2mo agoI've never done it but would be interested because it cuts out the burden of worrying about costs. Maybe I'm mistaken on energy cost here. There's a constant raincloud that follows me around regarding limits, and it would be nice to shake that. I've been able to accomplish incredible feats (for myself) since GPT-4, so model intelligence is secondary.
- eigenspace 2mo agoI think if there's going to be advantages to making smaller, more targeted models, those advantages will probably come from targeting specific domains, not from targeting specific languages. I think that if an LLM can't abstract over the differences between Python and C++, it probably will have an even harder time abstracting over the differences between writing code that manages a webserver, and writing code that does aerodynamic simulations.
- Archit3ch 2mo ago> 64GB Macbook pro with an M5 chip costs more than 4k Euros in Germany Sure, if you want the latest and almost* greatest. You can pick up an M1 Max 64GB for ~1k. * I guess 128GB also exists
- drob518 2mo agoThe machines that can run this are pricey, but not beyond a high end developer machine.
- ComputerGuru 2mo agoThere is no good reason to believe language-specific models are going to be any meaningfully smaller, just worse. Same as English-only models vs those trained on a multilingual corpus.
- Flere-Imsaho 2mo agoComing from the PC games industry in the 90s and early 2000s, it was a struggle to run some of the games on release. 90%* of people wouldn't be able to play the AAA games on release (think Crysis, etc). This period of local LLMs reminds me of that time, whereby the hardware just isn't there yet. Give it time, and the prices will drop. * total guess
- cube00 2mo agoAssuming we can even get the hardware in the first place, it might not even be possible for consumers to buy it at any price if it sells out through "agreements" made years in advance https://news.ycombinator.com/item?id=47045459 https://news.ycombinator.com/item?id=47045459
- delicious_apple 2mo agoI am running it on a single RTX 3090 (24GB VRAM). Some folks on Reddit are having the same experience: https://www.reddit.com/r/LocalLLaMA/comments/1vkm42m/muse_glimmer_actually_fits_on_a_single_rtx_3090/ https://www.reddit.com/r/LocalLLaMA/comments/1vkm42m/muse_gl... It uses an order of magnitude less VRAM at longer contexts which is a huge advantage over Qwen 3.6 27B
- r0b05 2mo agoSeems like that's the tradeoff with this model. Close to 27b intelligence while using less vram.