18 ms·
Ask HN: How close are we to local LLMs being useful? What's the impact?
Feels to me like local models are an under-covered aspect of this whole AI boom.
If everything improves over time, at some point a good chunk of tasks won’t need to be done in data centers or be subject to the whims of a few frontier AI labs.
How close are we to that? Or is my thinking flawed?
- buffer_overlord 4mo agountil its cheaper to train and infer than 100k gpu data centers...i doubt it will ever compete.
- tylerhackbart 4mo ago[flagged]
- PaulHoule 4mo agoI do classification with SLMs and for my tasks when I have a few thousand samples the frontier models in zero-shot and few-shot modes are embarassingly bad in comparison.
- david927 4mo agoI think we're past that point; they're absolutely useful already for a lot of tasks. I think it's about costs, convenience, and benefits of a frontier model for what you're doing.
- AbstractH24 4mo agoWhich ones are most useful? Any suggestions on where to go to start exploring this world?
- lgl 4mo agoA good place to browse is the LocalLLaMa subreddit. [0] A good software to start is LM Studio [1]. Another popular alternative is Ollama [2]. A better software when you're used to it all is llama.cpp as it's usually a bit faster and more frequently updated [3]. A good place to get models is HuggingFace, particularly the Unsloth models [4] Most popular models lately to run on "regular" gaming PC's, workstations, Macs etc are: Qwen 3.5 9b, Qwen 3.6 35B-A3B, Qwen 3.6 27B, Gemma 4. But there are hundreds or thousands of other models and different quantizations, finetunes, etc, etc. Have fun :) [0] https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ [1] https://lmstudio.ai/ https://lmstudio.ai/ [2] https://ollama.com/ https://ollama.com/ [3] https://github.com/ggml-org/llama.cpp https://github.com/ggml-org/llama.cpp [4] https://huggingface.co/unsloth/collections https://huggingface.co/unsloth/collections
- segmondy 4mo agoLocal LLMs have been useful since 2024. If you don't know this then you are just far behind. Catch up!
- riponcm 4mo ago[flagged]
- montfort 3mo agoTake a look at the AMD Ryzen AI Halo Developer Platform with a Ryzen AI Max+ 395 processor. These systems, with 128GB of unified memory and their specialized processors, deliver greater performance and inference power than Apple's Mac Studio. This already allows you to run fairly decent models for personal classification and coding tasks. I think for complex design needs you'll still require frontier models that can only be run in large data centers, but much of the underlying work could already be done on-premises.
- tae0086 3mo ago[dead]