Did you check your GPU utilization? Typically when it runs that way it runs on the CPU, not the GPU. Are you sure you're actually offloading any work to the G

Did you check your GPU utilization? Typically when it runs that way it runs on the CPU, not the GPU. Are you sure you're actually offloading any work to the GPU? At least with llama.cpp, there is no 'partially put a layer' into the GPU. Either you do, or you don't. You pick the number of layers. If the model is too big, the layers won't fit and it can't run at all. The llama.cpp `main` executable will tell you in it's debug information when you use the -ngl flag; see https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md#additional-options https://github.com/ggerganov/llama.cpp/blob/master/examples/... It's also possible you're running (eg. if you're using ollama) and quantized version of the model which reduces the memory requirements and quality of the model outputs.