4 ms·

I'll probably get stoned for asking here, but... since you seem knowledgeable on the subject: I just got llama3.1-8b (standard and instruct). However, I cannot

by eth0up 2y ago

I'll probably get stoned for asking here, but... since you seem knowledgeable on the subject: I just got llama3.1-8b (standard and instruct). However, I cannot do anything with it on my current hardware. Can you recommend the best AI model that I: 1) can self host 2) run on 16GB ram with no dedicated graphics card and an old intel i5 3) use on Debian without installing a bunch of exo-repo mystery code? Any recommendation, directly or semi related would be appreciated - I'm doing my 'research' but haven't made much progress nor had any questions answered.

arcanemachiner 2y ago
Setting up Ollama via Docker was the easiest way for me to get up and running. Not 100% sure if it fits your constraints, but highly recommended.
- programd 2y ago
  Another option is to download and compile llama.cpp and you should be able to run quantized models at an acceptable speed. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Also, if you can spend the $60 and buy another 32GB of RAM, this will allow you to run the 30GB models quite nicely.
  - eth0up 2y ago
    Unfortunately motherboard is capped at 16Gb ram
smokel 2y ago
Running LLMs on that kind of hardware will be very slow (expect responses with only a few words per second, which is probably pretty annoying). LM Studio [1] makes it very easy to run models locally and play with them. Llama 3.1 will only run in quantized form with 16GB RAM, and that cripples it quite badly, in my opinion. You may try Phi-3 Mini, which has only 3.8B weights and can still do fun things. [1] https://lmstudio.ai/ https://lmstudio.ai/
- eth0up 2y ago
  Much appreciated. Thanks for this!
  - deleted 2y ago
    [deleted]
- wkat4242 2y ago
  I don't find llama3.1 noticeably worse on 8 bit integer quantised than the original fp16 to be honest. It's also a lot faster. Of course even then you're not going to reach the whole 128k context window on 16GB but if you don't need that it works great.

by eth0up 2y ago

arcanemachiner 2y ago

Setting up Ollama via Docker was the easiest way for me to get up and running. Not 100% sure if it fits your constraints, but highly recommended.

programd 2y ago
Another option is to download and compile llama.cpp and you should be able to run quantized models at an acceptable speed. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Also, if you can spend the $60 and buy another 32GB of RAM, this will allow you to run the 30GB models quite nicely.
- eth0up 2y ago
  Unfortunately motherboard is capped at 16Gb ram

smokel 2y ago

Running LLMs on that kind of hardware will be very slow (expect responses with only a few words per second, which is probably pretty annoying). LM Studio [1] makes it very easy to run models locally and play with them. Llama 3.1 will only run in quantized form with 16GB RAM, and that cripples it quite badly, in my opinion. You may try Phi-3 Mini, which has only 3.8B weights and can still do fun things. [1] https://lmstudio.ai/ https://lmstudio.ai/

eth0up 2y ago
Much appreciated. Thanks for this!
- deleted 2y ago
  [deleted]
wkat4242 2y ago
I don't find llama3.1 noticeably worse on 8 bit integer quantised than the original fp16 to be honest. It's also a lot faster. Of course even then you're not going to reach the whole 128k context window on 16GB but if you don't need that it works great.