3 ms·
It has partial gpu acceleration if you compile it with LLAMA_CUBLAS or LLAMA_CLBLAST They really have come a long way since... A few weeks ago. Using cublas w
by azeirah 3y ago
It has partial gpu acceleration if you compile it with LLAMA_CUBLAS or LLAMA_CLBLAST
They really have come a long way since... A few weeks ago.
Using cublas with my 1080ti results in a 52% speedup compared to cpu-only. Vram usage is very minimal.