4 ms·

I might be missing something. The article asks me to run a bash script on windows. I assume this would still need to be run manually to access GPU resources et

by maxlin 3y ago

I might be missing something. The article asks me to run a bash script on windows. I assume this would still need to be run manually to access GPU resources etc, so can someone illuminate what is actually expected for a windows user to make this run? I'm currently paying 15$ a month in a personal translation/summarizer project's ChatGPT queries. I run whisper (const.me's GPU fork) locally and would love to get the LLM part local eventually too! The system generates 30k queries a month but is not super-affected by delay so lower token rates might work too.

nomel 3y ago
Windows has supported linux tools for some time now, using WSL: https://learn.microsoft.com/en-us/windows/wsl/about https://learn.microsoft.com/en-us/windows/wsl/about No idea if it will work, in this case, but it does with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/103 https://github.com/ggerganov/llama.cpp/issues/103
- maxlin 3y ago
  I know (should have included in my earlier response but editing would've felt weird) but I still assume one should run the result natively, so am asking if/where there's some jumping around required. Last time I tried running an LLM I tried wsl&native both on 2 machines and just got lovecraftian-tier errors so waiting if I'm missing something obvious before going down that route again
Charlieholtz 3y ago
Thanks for pointing this out — we should've pointed out the script needs to be run on WSL. I added a note to the post to clarify (I work at Replicate). Also, you don't need a GPU to run this script! It builds and runs llama.cpp https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
mattstir 3y ago
The bash script is downloading llama.cpp, a project which allows you to run LLaMA-based language models on your CPU. The bash script then downloads the 13 billion parameter GGML version of LLaMA 2. The GGML version is what will work with llama.cpp and uses CPU for inferencing. There are ways to run models using your GPU, but it depends on your setup whether it will be worth it. I would highly recommend looking into the text-generation-webui project (https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui). It has a one-click installer and very comprehensive guides for getting models running locally and where to find models. The project also has an "api" command flag to let you use it like you might use a web-based service currently.
shortrounddev2 3y ago
check my comment elsewhere in the thread, I wrote build instructions for windows + Nvidia GPUs

by maxlin 3y ago

nomel 3y ago

Windows has supported linux tools for some time now, using WSL: https://learn.microsoft.com/en-us/windows/wsl/about https://learn.microsoft.com/en-us/windows/wsl/about No idea if it will work, in this case, but it does with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/103 https://github.com/ggerganov/llama.cpp/issues/103

maxlin 3y ago
I know (should have included in my earlier response but editing would've felt weird) but I still assume one should run the result natively, so am asking if/where there's some jumping around required. Last time I tried running an LLM I tried wsl&native both on 2 machines and just got lovecraftian-tier errors so waiting if I'm missing something obvious before going down that route again

Charlieholtz 3y ago

Thanks for pointing this out — we should've pointed out the script needs to be run on WSL. I added a note to the post to clarify (I work at Replicate). Also, you don't need a GPU to run this script! It builds and runs llama.cpp https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp

mattstir 3y ago

The bash script is downloading llama.cpp, a project which allows you to run LLaMA-based language models on your CPU. The bash script then downloads the 13 billion parameter GGML version of LLaMA 2. The GGML version is what will work with llama.cpp and uses CPU for inferencing. There are ways to run models using your GPU, but it depends on your setup whether it will be worth it. I would highly recommend looking into the text-generation-webui project (https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui). It has a one-click installer and very comprehensive guides for getting models running locally and where to find models. The project also has an "api" command flag to let you use it like you might use a web-based service currently.

shortrounddev2 3y ago

check my comment elsewhere in the thread, I wrote build instructions for windows + Nvidia GPUs