I'm doing that with a 12GB card, ollama supports it out of the box. For some reason, it only uses around 7GB of VRAM, probably due to how the layers are schedu

I'm doing that with a 12GB card, ollama supports it out of the box. For some reason, it only uses around 7GB of VRAM, probably due to how the layers are scheduled, maybe I could tweak something there, but didn't bother just for testing. Obviously, perf depends on CPU, GPU and RAM, but on my machine (3060 + i5-13500) it's around 2 t/s.