3 ms·
7b can run fine on consumer hardware if you quantize to 4bit. I get around 50-100 token/s on 7b q4 models on older 8gb gpus I have lying around, which is plenty
by eyegor 3y ago
7b can run fine on consumer hardware if you quantize to 4bit. I get around 50-100 token/s on 7b q4 models on older 8gb gpus I have lying around, which is plenty for chat. I don't know of any good resource for "consumer hardware leaderboard" though. In general:
1.3b is about the max you can get for 4gb vram
7b will work on 8gb vram if you quantize
13b will work on 16gb vram if you quantize
33b can work on 24gb if you quanitze
If you lack vram but have ram and infinite patience you can do offloading shenanigans with cpu but it tanks performance to unusable levels in my experience.
- instagib 3y agoConsidering quantizing takes a while, care to list some settings guide or a link for people to setup? I use Phind_34B for a 24GB gpu. When in use I typically set gpu memory limit to 22.5GB. Quantization: 1hr 3min. 4bit, group_size=32, desc=true, damp=0.01, trust_remote = false, dtype torch.float16, use_triton = false.
- eyegor 3y agoI'd defer to the llama.cpp or exllamav2 repos for quantizing help, I just use their guides myself. For practical use, I'd say searching for the quantized model on huggingface.co is the way to go. Chances are high that someone else already did what you're interested in and it's a lot faster to download a pre quantized model. Thebloke, turboderp, lonestriker, bartowski. Between these 4 users on huggingface you can find pretty much every common model already quantized.
- avereveard 3y agoWhat do you use to run them? My 3060 is generating just 40token per second with llama.cpp
- eyegor 3y agoI use either llama.cpp or exllamav2 as the backend, but speed is mostly dependent on how many layers fit in your gpu. I'm not sure which gpu I was thinking of for that 50-100 tok/s, on my 2070 I can only get around 15 tok/s on 7b models. I have to use 1.3b models to get to 100 tok/s on this gpu.