3 ms·
Off topic, but I’m very interested in local LLMs. Could you point me in the right direction, both hardware specs and models?
by BenFranklin100 2y ago
Off topic, but I’m very interested in local LLMs. Could you point me in the right direction, both hardware specs and models?
- thrownblown 2y agohttps://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/SillyTavernAI/ https://www.reddit.com/r/SillyTavernAI/
- BenFranklin100 2y agoThanks to both of you!
- doctoboggan 2y agoIn general for local LLMs, the more memory the better. You will be able to fit larger models in RAM. The faster CPU will give you more tokens/second, but if you are just chatting with a human in the loop, most recent M series macs will be able to generate tokens faster than you can read them.
- int_19h 2y agoThat also very much depends on model size. For 70B+ models, while the tok/s are still fast enough for realtime chat, it's not going to be generating faster than you can read it, even on Ultra with its insane memory bandwidth.
- noman-land 2y agoGet as much RAM as you can stomach paying for.
- touristtam 2y agoHave a look at ollama? I think there is a vscode extension to hook into local LLM if you are so inclined: https://ollama.com/blog/continue-code-assistant https://ollama.com/blog/continue-code-assistant