3 ms·
You need suitable hardware (ideally a 3090, 4090 or an Apple M device with a decent amount of mem). Then set up software - ollama for easy mode (but less contr
by Havoc 3y ago
You need suitable hardware (ideally a 3090, 4090 or an Apple M device with a decent amount of mem).
Then set up software - ollama for easy mode (but less control) or text-generation-webui for more control.
After that you can just try models. The subreddit /r/localllama has whatever is flavour of the week. The Mixtral model at like Q3 quantization is probably a good starting point
- daghamm 3y agoWith ROCm 6 I think new-ish AMD GPUs are pretty good too. https://ollama.com/blog/amd-preview https://ollama.com/blog/amd-preview With that said, I dont think you need anything special to run LLMs these days. I can run 7B models on a 4 year old AMD or Intel CPU (no GPU), for programming tasks.
- Havoc 3y agoIndeed...bought AMD shares on the bet that they'll catch up. However this is still true only in a fairly narrow space - inference - once you stray off that very narrow path they're still a fair bit behind
- FezzikTheGiant 3y agoSo I can run local LLMs on my apple M1 air?
- tharmas 3y agoYes you can. Download LM studio and then search for a model such as Mistral-7B-Instruct-v0.2. LM studio will suggest variants of that model that suit your hardware to download. Here's a demonstration: https://www.youtube.com/watch?v=VXHryjPu52k https://www.youtube.com/watch?v=VXHryjPu52k