4 ms·
Better privacy might be running Llama2 locally, offline. If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or loca
by jankovicsandras 3y ago
Better privacy might be running Llama2 locally, offline.
If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or locally. The !s are for Colab, remove them for local terminal. The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards.
! git clone https://github.com/ggerganov/llama.cpp.git
! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q8_0.gguf" -P llama.cpp/models
! cd llama.cpp && make
! ./llama.cpp/main -m ./llama.cpp/models/codellama-7b.Q8_0.gguf --color --ctx_size 2048 -n -1 -ins -b 256 --top_k 10000 --temp 0.2 --repeat_penalty 1.1 -t 8
- lardo 3y agohttps://huggingface.co/TheBloke/CodeLlama-7B-GGUF https://huggingface.co/TheBloke/CodeLlama-7B-GGUF describes codellama-7b.Q8_0.gguf as "very large, extremely low quality loss - not recommended"
- jmorgan 3y agoQ4_0 is often mentioned as being the "tried and true" quantization level to try first. I've heard folks have had good results with 3-bit quantization (Q3_K_M) as well
- jsemrau 3y agojust tried it and its not good. I think one of the better feedback was asking for play an adventure in the style of Space Quest 1
- Patrick_Devine 3y agoOr, for the easy way of doing it, just install Ollama and run: `ollama run codellama:7b`. If you want the q8 version (which you probably don't), it's `ollama run codellama:7b-code-q8_0`.
- rvz 3y agoOr if you're on macOS just hit 'Download' and use Ollama locally. [0] [0] https://ollama.ai/download https://ollama.ai/download