Llama.cpp has Vulkan text generation, so you can use the GPU without any special drivers. It's at least 10x faster than CPU generation and turns small models (

Llama.cpp has Vulkan text generation, so you can use the GPU without any special drivers. It's at least 10x faster than CPU generation and turns small models (up to 7B parameters) into an experience as fast as any of the commercial services.