3 ms·
PSA: If you're using oobabooga/text-generation-webui then you need to: 1. Re-install llama-cpp-python with support for CUBLAS: CMAKE_ARGS="-DLLAMA_CUBLAS=o
by rahimnathwani 3y ago
PSA:
If you're using oobabooga/text-generation-webui then you need to:
1. Re-install llama-cpp-python with support for CUBLAS:
CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir --force-reinstall
2. Launch the web UI with the --n-gpu-layers flag, e.g.
python server.py --model gpt4-x-vicuna-13B.ggml.q5_1.bin --n-gpu-layers 24