34 ms·
I see all these LLM posts about if a certain model can run locally on certain hardware and I don’t get it. What are you doing with these local models that run
by testing22321 7mo ago
I see all these LLM posts about if a certain model can run locally on certain hardware and I don’t get it.
What are you doing with these local models that run at x tokens/sec.
Do you have the equivalent of ChatGPT running entirely locally? What do you do with it? Why? I honestly don’t understand the point or use case.
- samuel 7mo agoChat is certainly an option, but the real deal are agents, which have access to way more sensitive information.
- testing22321 7mo agoThanks. What do you do with such an agent? What is the use case?
- dec0dedab0de 7mo agomost of the llm tooling can handle different models. Ollama makes it easy to install and run different models locally. So you can configure aider or vscode or whatever you're using to connect to chatgpt to point to your local models instead. None of them are as good as the big hosted models, but you might be surprised at how capable they are. I like running things locally when I can, and I also like not worrying about accidentally burning through tokens. I think the future is multiple locally run models that call out to hosted models when necessary. I can imagine every device coming with a base model and using loras to learn about the users needs. With companies and maybe even households having their own shared models that do heavier lifting. while companies like openai and anhtropic continue to host the most powerful and expensive options.
- roboror 7mo agoWhat models have you found capable? I was recently recommended Qwen3 Coder Next and I did not find it very successful. I have a good amount of VRAM/RAM so would love to run something locally.
- testing22321 7mo agoThanks. I still don’t understand. What are you using this long you’re running locally to actually do? What is the use case?
- svachalek 7mo ago1. There are small local models that have the capabilities of frontier models a year ago 2. They aren't harvesting your data for government files or training purposes 3. They won't be altered overnight to push advertising or a political agenda 4. They won't have their pricing raised at will 5. They won't disappear as soon as their host wants you to switch
- testing22321 7mo agoThanks. I understand that. What are you doing with it? Why do you want it?
- jkl5xx 7mo agoGood points. What local models have you found work best for your use cases? I feel like if we get to opus 4.6 level intelligence running on local hardware, we’re in the clear for a lot of day to day use cases.
- derangedHorse 7mo agoQwen3.5 is like an old version of ChatGPT and I can use it the same way I used GPT4 — writing emails, reading documentation and answering questions about it, reviewing code, answering trivia, etc.