3 ms·
How? 24GB VRAM can only handle 30B 4bit. Are you offloading some layers to the CPU? That kills performance AFAIK.
by redox99 3y ago
How? 24GB VRAM can only handle 30B 4bit. Are you offloading some layers to the CPU? That kills performance AFAIK.