3 ms·
They really aren't. 30B models are much smarter, and work fine with 4 bit quantization on a 24GB GPU. On a headless system you can get the full 2048 context siz
by cypress66 3y ago
They really aren't. 30B models are much smarter, and work fine with 4 bit quantization on a 24GB GPU. On a headless system you can get the full 2048 context size. On a desktop around 1500 I think.