3 ms·

Ask HN: Recommendations for Running LLMs Locally

3 points by fallinditch 2y ago

Hi folks I'm looking for hardware recommendations for running LLMs locally. Specifically, best bang for bucks at the following price points: $2,000 $5,000 $10,000 What more could you do with a 10k budget than with a 5k budget? Thank you

vunderba 2y ago
Previous thread from just a few days ago (5k budget): https://news.ycombinator.com/item?id=42853287 https://news.ycombinator.com/item?id=42853287
BMc2020 2y ago
I have an old dell latitude 3420 with 16 gig of ram. It's running Pop!_OS. I used Alpaca to load the Mistral(7b) model. I ask it questions and it answers. Didn't cost anything. On a different computer running Cinnamon Mint I used GPT4ALL to load a Llama 3 8b model. I use it to read a bunch of local docs for me. It also didn't cost anything.
- eth0up 2y ago
  Would you please describe the basics of this process? I am interested in doing this and my hardware is similarly limited. No dedicated GPU, but 24g ram and solid CPU.
  - nejsjsjsbsb 2y ago
    docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama docker exec -it ollama ollama run llama3.2:1b Have a chat at the CLI Then try bigger models if that is too fast for you :)
  - BMc2020 2y ago
    Alpaca is in the Pop!_shop (what is called the store on most other distros) so it's just a matter of clicking install. Inside it it has a list of several models, I downloaded a small model, but even a small model is very big, 4 Gigs. Some are over 100GB. When one is downloaded just click on new chat. https://github.com/Jeffser/Alpaca https://github.com/Jeffser/Alpaca GPT4ALL on the cinnamon mint box I first installed Easy Flatpak then installed it from there. That I had a little different goal, I had a lot of pdf and epub books I wanted to index. I downloaded a small model again, a 4.34G Llama 3 8B Instruct. In the local docs I put one pdf 'Github for Dummies' and asked it to index. It took about 30 minutes IIRC. (don't start with too many, it could be days indexing. You can add docs later and re-index so you can build your database slowly) Then instead of looking up in the book how to revert a commit, you chat with the model and ask it how to revert a commit. My take is the model is the human language interface for queries, and that's all. Bear in mind almost everyone knows more about this me, I just wanted to see what all the fuss was about. You can also do similar what I did with gpt4all online for free as long as you have a google account. Go to notebooklm dot google dot com and start uploading your documents.
    - eth0up 2y ago
      Thanks for the info.
- fallinditch 2y ago
  I like this approach: use a bunch of LLMs on various machines. SomeOddCodeGuy in Reddit shared his very cool multi LLM setup [1]. I quote: > I have 2 "assistants" set up, with the intention to later add a third. Each assistant is essentially built to be an advanced "rubber duck" (as in the rubber duck programming method where you talk through a problem to an inanimate object and it helps you solve this problem). Each assistant is built entirely to talk through problems with me, of any kind, and help me solve them by challenging me, answering my questions, or using a specific set of instructions on how to think through issues in unique ways. Each assistant is built to be different, and thus solve things differently. Each assistant is made up of multiple LLMs. Some examples would be: A responder model, which does the talking A RAG model, which I use for pulling data from the offline wikipedia api for factual questions A reasoning model, for thinking through a response before the responder answers A coding model, for handle code issues and math issues. See his post at https://www.reddit.com/r/LocalLLaMA/comments/1i1kz1c/sharing_my_unorthodox_home_setup_and_how_i_use/ https://www.reddit.com/r/LocalLLaMA/comments/1i1kz1c/sharing...
deleted 2y ago
[deleted]
sandreas 2y ago
Well, this might be unpopular but for the 2000$ price point it may be the macOS ecosystem. The most important part to run LLMs might be the VRAM-size. Since modern Macs use unified memory you simplified get RAM=VRAM, so a 64GB MacBook Pro might outperform an NVidia 4090 in LLMs. However the specialized cards from nvidia have more VRAM and are generally faster. The LLM of your choice should fit into vram, so if you only need 21gb, one 4090 will do the job

3 points by fallinditch 2y ago

vunderba 2y ago

Previous thread from just a few days ago (5k budget): https://news.ycombinator.com/item?id=42853287 https://news.ycombinator.com/item?id=42853287

BMc2020 2y ago

I have an old dell latitude 3420 with 16 gig of ram. It's running Pop!_OS. I used Alpaca to load the Mistral(7b) model. I ask it questions and it answers. Didn't cost anything. On a different computer running Cinnamon Mint I used GPT4ALL to load a Llama 3 8b model. I use it to read a bunch of local docs for me. It also didn't cost anything.

eth0up 2y ago
Would you please describe the basics of this process? I am interested in doing this and my hardware is similarly limited. No dedicated GPU, but 24g ram and solid CPU.
- nejsjsjsbsb 2y ago
  docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama docker exec -it ollama ollama run llama3.2:1b Have a chat at the CLI Then try bigger models if that is too fast for you :)
- BMc2020 2y ago
  Alpaca is in the Pop!_shop (what is called the store on most other distros) so it's just a matter of clicking install. Inside it it has a list of several models, I downloaded a small model, but even a small model is very big, 4 Gigs. Some are over 100GB. When one is downloaded just click on new chat. https://github.com/Jeffser/Alpaca https://github.com/Jeffser/Alpaca GPT4ALL on the cinnamon mint box I first installed Easy Flatpak then installed it from there. That I had a little different goal, I had a lot of pdf and epub books I wanted to index. I downloaded a small model again, a 4.34G Llama 3 8B Instruct. In the local docs I put one pdf 'Github for Dummies' and asked it to index. It took about 30 minutes IIRC. (don't start with too many, it could be days indexing. You can add docs later and re-index so you can build your database slowly) Then instead of looking up in the book how to revert a commit, you chat with the model and ask it how to revert a commit. My take is the model is the human language interface for queries, and that's all. Bear in mind almost everyone knows more about this me, I just wanted to see what all the fuss was about. You can also do similar what I did with gpt4all online for free as long as you have a google account. Go to notebooklm dot google dot com and start uploading your documents.
  - eth0up 2y ago
    Thanks for the info.
fallinditch 2y ago
I like this approach: use a bunch of LLMs on various machines. SomeOddCodeGuy in Reddit shared his very cool multi LLM setup [1]. I quote: > I have 2 "assistants" set up, with the intention to later add a third. Each assistant is essentially built to be an advanced "rubber duck" (as in the rubber duck programming method where you talk through a problem to an inanimate object and it helps you solve this problem). Each assistant is built entirely to talk through problems with me, of any kind, and help me solve them by challenging me, answering my questions, or using a specific set of instructions on how to think through issues in unique ways. Each assistant is built to be different, and thus solve things differently. Each assistant is made up of multiple LLMs. Some examples would be: A responder model, which does the talking A RAG model, which I use for pulling data from the offline wikipedia api for factual questions A reasoning model, for thinking through a response before the responder answers A coding model, for handle code issues and math issues. See his post at https://www.reddit.com/r/LocalLLaMA/comments/1i1kz1c/sharing_my_unorthodox_home_setup_and_how_i_use/ https://www.reddit.com/r/LocalLLaMA/comments/1i1kz1c/sharing...

deleted 2y ago

[deleted]

sandreas 2y ago

Well, this might be unpopular but for the 2000$ price point it may be the macOS ecosystem. The most important part to run LLMs might be the VRAM-size. Since modern Macs use unified memory you simplified get RAM=VRAM, so a 64GB MacBook Pro might outperform an NVidia 4090 in LLMs. However the specialized cards from nvidia have more VRAM and are generally faster. The LLM of your choice should fit into vram, so if you only need 21gb, one 4090 will do the job