4 ms·
You don't need so many layers of stuff (or API keys, signups, or other nonsense). Llama.cpp (to serve the model) + the Continue VS Code extension are enough.
by npsomaratna 3y ago
You don't need so many layers of stuff (or API keys, signups, or other nonsense).
Llama.cpp (to serve the model) + the Continue VS Code extension are enough.
The rough list of steps to do so are:
Part A: Install llama.cpp and get it to serve the model:
--------------------------------------------------------
1. Install the llama.cpp repo and run make.
2. Download the relevant model (e.g. wizardcoder-python-34b-v1.0.Q4_K_S.gguf).
3. Run the llama.cpp server (e.g., ./server -t 8 -m models/wizardcoder-python-34b-v1.0.Q4_K_S.gguf -c 16384 --mlock).
4. Run the OpenAI like API server [also included in llama.cpp] (e.g., python ./examples/server/api_like_OAI.py).
Part B: Install Continue and connect it to llama.cpp's OpenAI like API:
-----------------------------------------------------------------------
5. Install the Continue extension in VS Code.
6. In the Continue extension's sidebar, click through the tutorial and then type /config to access the configuration.
7. In the Continue configuration, add "from continuedev.src.continuedev.libs.llm.ggml import GGML" at the top of the file.
8. In the Continue configuration, replace lines 57 to 62 (or around) with:
models=Models(
default=GGML(
max_context_length=16384,
server_url="http://localhost:8081"
)
),
9. Restart VS Code, and enjoy!
You can access your local coding LLM through the Continue sidebar now.
- vanillax 3y agowodner if you can pair with https://github.com/getumbrel/llama-gpt https://github.com/getumbrel/llama-gpt
- noiv 3y agoThx. Where can I send flowers to?
- ignoramous 3y agoTo any person you're in a position to be kind to.
- redox99 3y agoIs there a way to make it work with ooba+exllama? (much faster than llamacpp)
- thelastparadise 3y agoYou should be able to turn on the API in booba: https://github.com/oobabooga/text-generation-webui#api https://github.com/oobabooga/text-generation-webui#api
- redox99 3y agoBut that API isn't OpenAI compatible AFAIK
- k4rli 3y agoThanks, works nicely and easy to set up. Is it possible to use GPU for this? With R9 7900x and 32GB RAM it takes 15-30sec to generate response. I have a 6900XT which might be more suited for this.
- npsomaratna 3y agoYes. In the llama.cpp server command, specify the number of layers you'd like offloaded to your GPU via the -ngl parameter, e.g.: ./server -t 8 -m models/wizardcoder-python-34b-v1.0.Q4_K_S.gguf -c 16384 --mlock -ngl 60 (You might need to play around with the number of layers.) [Edit: make sure to compile llama.cpp with GPU support first, e.g., "make clean && LLAMA_CUBLAS=1 make -j"]
- tayo42 3y agoOne of the most annoying things about learning ai/ml for me right now is how much of this stuff is hidden behind people's comlanies and projects with to many emojis. Like I can't find simple straight foward solutions or content that isn't tied back to a company.
- jfoucher 3y agoI'm a complete beginner regarding this stuff, so if I may ask, how would I go about downloading the relevant model (e.g. wizardcoder-python-34b-v1.0.Q4_K_S.gguf) I checked on Hugging face but all I got was a bunch of .bin files... Thanks.
- rpgwaiter 3y agoGoogle the filename + "torrent download"
- npsomaratna 3y agoDo a search on the HuggingFace models page, e.g.: https://huggingface.co/models?sort=trending&search=wizardcoder+gguf https://huggingface.co/models?sort=trending&search=wizardcod...
- jfoucher 3y agoThanks, I managed to convert what I had downloaded with the convert.py script in llama.cpp.