4 ms·
The only info I can provide is the table I've seen on: https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama where it states one needs "32 GB
by Ms-J 3y ago
The only info I can provide is the table I've seen on: https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama where it states one needs "32 GB to run the 13B models." I would assume you may need a GPU for this.
Related, could someone please point me in the right direction on how to run Wizard Vicuna Uncensored or Llama2 13B locally in Linux? I've been searching for a guide and have not found what I need for a beginner like myself. In the Github I referenced the download is only for Mac at the time. I have a Macbook Pro M1 I can use though it's running Debian.
Thank you.
- hdjfkfbfbr 3y agoHmmm I ran llama2 ggml q4 in 6gb ram with llama.CPP on my laptop.
- Ms-J 3y agoI very much do appreciate your comment and will look into into llama.cpp. Was it from here: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Do you have a guide that you followed and could link it to me or was it just from prior knowledge? Also, do you know if I could run the Wizard Vicuna on it? That model isn't listed on the above page.
- hdjfkfbfbr 3y agoGlad to be of help. Yea that is the repo. https://replicate.com/blog/run-llama-locally https://replicate.com/blog/run-llama-locally I found that guide here on hn. I run it cpu only with 16 threads but yeah perf is good enough. BTw my 6gb figure is me.measuring from htop so llama2 is likely less.
- Ms-J 3y agoThanks for the starting point. I'll give an update if I'm able to successfully run the other models. I hope it could help the community.
- hdjfkfbfbr 3y agoNot sure about vicuna myself
- singhrac 3y agoThis code runs Llama2 quantized and unquantized in a roughly minimal way: https://github.com/srush/llama2.rs https://github.com/srush/llama2.rs (though extracting the quantized 70B weights takes a lot of RAM). I'm running the 13B quantized model on ~10-11GB of CPU memory.
- Ms-J 3y agoFrom what I gather, this is a Rust implementation that runs Llama2. Can it run any other models like the ones I'm having trouble finding info about?
- Patrick_Devine 3y agoYou can run `ollama run wizard-vicuna-uncensored:13b` and it should pull and run it. For llama2 13b, it's `ollama run llama2:13b`. I haven't seen the 13b uncensored version yet. There's a complete list of models at https://gist.github.com/mchiang0610/b959e3c189ec1e948e4f6a1f737a1fc5 https://gist.github.com/mchiang0610/b959e3c189ec1e948e4f6a1f... We'll have a better way to browse these soon.