Assuming an nvidia GPU and requisite system memory, use llama.cpp compiled with cublas support, then run with the -ngl [n layers] option. You'll need a model qu

Assuming an nvidia GPU and requisite system memory, use llama.cpp compiled with cublas support, then run with the -ngl [n layers] option. You'll need a model quantized after May 12 to work with this. The smallest GPU-only 7B 4-bit model requires 8GB VRAM, so it's either do CPU only or use the GPU offload above.