3 ms·
> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. Cloud models are (much) faster, they don't consume so much po
by pizza234 4mo ago
> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it.
Cloud models are (much) faster, they don't consume so much power/generate heat, they have much bigger (LLM) context, they're much more precise and they have a much wider (engineering) context of the given problem.
Except privacy and use cases that are blocked by cloud models (e.g. reverse engineering), local LLMs are currently an expensive toy.
When I try to program with a local LLM (I'm on a 32/128 GB system), I end up wasting time compared to a cloud LLM.
- dofm 4mo agoAgain, I would not argue against any of this. And I can't say that I won't switch to openrouter (even just for the same models) at some point. But one of the things I have found about my own process learning is that some lessons only come to you when you make yourself available to them. And if that means doing things the difficult way, that is what you should do.
- wahnfrieden 4mo agoDifficult... and wastefully expensive
- dofm 4mo agoI mean, it's a (secondhand) computer I bought for other tasks (processing very large photos, compiling large apps quickly). It's running all the time. It can also run LLMs when I want to. The rest of my life is ultra-frugal so I am relaxed about this.
- _puk 4mo agoDon't bite. You're right. Having spent a good weekend learning how to perform latent-steering through playing with pytorch and a local Gemma4 model, there is no way I could have groked any of that in the the way I did without hands on time. This is on an M3 Max 36GB I've had for a couple of years. No further outlay needed.
- monkmartinez 4mo agoMy thinking is totally aligned with yours, perhaps its because I am trying to do a second act at almost 50 from blue-collar to white collar office work. I have no formal degree, but I have been hobby programming for 20 years. I have made a habit of "letting myself be available to all lessons"... the localllama group has made this journey really fun if nothing else. I have learned an ABSOLUTE ton from this era!
- dofm 4mo agoI have been contemplating a move in the opposite direction because I have just been exhausted and depressed, so for me, really learning this stuff this way has been about managing those feelings, about a sense of pride and ownership of my processes. I don't know if it has changed my mind about a career change but as I am sure you can understand, I no longer feel like I am running away defeated. My very best wishes to you :-)
- sanderjd 4mo agoSeems like an investment into building expertise, which is likely to have high ROI in the future, rather than a wasteful cost.
- pizza234 4mo ago> Seems like an investment into building expertise Learning LLM internals != using them as daily tools (the latter was the topic of discussion). Shifting to the learning topic: no need for expensive hardware. Even a 16 GB GPU will do, and that's (relatively) cheap and available.
- moffkalast 4mo agoPeople pay thousands for model trains, everyone needs a hobby.
- dofm 4mo agoTraining models vs modelling trains
- moffkalast 4mo agoAh yes, the EMD0E9-30B-Union-Pacific.gguf
- fragmede 4mo agoI'm sending Codex-gpt-5.5-cyber.stl to my printer right now!
- AlpacaJones 4mo agoThe key word there is 'currently'.
- smt88 4mo agoEconomies of scale are a fact of nature and aren’t going to be subverted in the future by even the most advanced local models
- kennywinker 4mo agoWhich is of course why, if you want to render 3d scenes to play a video game, you have to rent time on a mainframe system. I don’t see that changing ever - it’s just economies of scale! (sarcasm, btw)
- oceanplexian 4mo agoThings can get both more expensive and cheaper at scale, hence the term. For example (and relevant to AI) I can generate electricity on my roof at $0.20-25/kWh, batteries included. In California the electric utility can’t offer it cheaper than $0.30-0.50/kWh. Therefore at scale, electricity is actually more expensive. There are many such examples.
- sanderjd 4mo agoYeah, I think the fallacy here is the conflation of scale and centralization. Right now, there is way more scale in centralized AI than there is at the edge. But that could flip. I'd still probably put the probability that it will under 50%. But I'd also put it above zero!
- Dylan16807 4mo agoApples and Oranges. The utility uses a weird conflated fee that combines the price of the electricity and the price of connecting your house to the grid. If they split it up your marginal price per kWh would be much less.
- sanderjd 4mo ago
- deleted 4mo ago[deleted]
- deleted 4mo ago[deleted]
- psychoslave 4mo agoAnything done local will likely come at higher cost and at scale with less energy efficiency and commodity, with less possibility to fine tune engineer deeply on wider horizon of issues. That's never the point of keeping local alternatives though.
- dofm 4mo agoRight. For me this dates all the way back to installing Slackware 1.0 (0.99pl12!) on an offline 486SX rather than just using the internet-connected workstations in the lab. Here, I already had a Mac that was powerful enough to run a local LLM, so now I do, because I can.
- bogeholm 4mo ago> Cloud models […] don't consume so much power/generate heat I do realize the cloud is just someone else’s computer right? Power goes in, tokens and heat come out - just in another place
- actionfromafar 4mo agoThe cloud computers produce more tokens per watt. That said, if you have a computer at home running 24/7 for other reasons and you also can use it for some LLM work, why not.
- sanderjd 4mo ago> currently The interesting question is whether that gap will narrow, and if so, how much, and on what timescale. The exact answer to this question is not knowable, but if you are the kind of person who comes to a site called "hacker news", and you think there is a nonzero chance that the answer is that yes, the gap will narrow and this won't always be an expensive toy, then now seems like a pretty great time to get in the game and start exploring the capabilities.
- pizza234 4mo agoThe same applies to ray tracing; the question whether shadows can be rendered accurately and cheaply is also not knowable - but it's very unlikely that ray tracing will become as cheap as rasterization for the same quality target. LLMs are pretty much the same; they're inherently computationally expensive, even if optimizations are in continuous development.
- Abishek_Muthian 4mo agoI agree completely. I think local AI is best limited to purpose built SLMs; all this craze around running quantized coding LLMs has taken the attention off SLMs.
- icedchai 4mo agoSame. Local LLMs are fun to experiment with, but when I want generated code of a sufficient quality, I use a cloud LLM.
- Shorel 4mo agoFrom your post I can only perceive the instinct to pick a side, and trying to make sure it is the "winning side". But the truth is far more nuanced. I have acces to both, paid and local models, and even if slower, the local models have been far more educative about how these technologies are put together, and what is required for local computing to thrive again. Paid models will not suddenly disappear just because I play with glm-4.6 on Ollama. At the same time, my work pays the cloud subscription and I use the cloud models to perform the tasks my work requires. There's no need to choose one side.
- cyanydeez 4mo agothe downsides are also: It's underlying "alignment" can change at any time, it's non-determinism is manifold, because of the alignment and the entire aparatus behind it (currently claude's KV cache use is turning it into a $$$$ burning machine), it has zero customer service expectation, and they can ban you at any time. So if you're just putzing around, sure, cloud will be fine, but if you're building a workflow/business around these things, you're essentially making yourself an indentured servant.