4 ms·
Llama.cpp recently added partial GPU acceleration. Model dequantization as well as some BLAS operations have been moved to GPU. It runs a lot faster if you com
by azeirah 3y ago
Llama.cpp recently added partial GPU acceleration. Model dequantization as well as some BLAS operations have been moved to GPU.
It runs a lot faster if you compile with cuBLAS (nvidia) or clblast (other). GPU vram doesn't matter much since it doesn't offload the model to vram.