3 ms·
Is there a way to run any of these with only 4GB of VRAM?
by blendergeek 3y ago
Is there a way to run any of these with only 4GB of VRAM?
- washadjeffmad 3y agoAssuming an nvidia GPU and requisite system memory, use llama.cpp compiled with cublas support, then run with the -ngl [n layers] option. You'll need a model quantized after May 12 to work with this. The smallest GPU-only 7B 4-bit model requires 8GB VRAM, so it's either do CPU only or use the GPU offload above.
- blendergeek 3y agoThank you! I'll give it a try.