3 ms·

Is there any rough rules on what model sizes will work on various RAM sizes on consumer machines?

• washadjeffmad 3y ago
  For "full" context, typically: 7B: 8GB (4GB min for model only) 13B: 12GB (8GB min) 34B: 28GB (18GB min) 70B: 60GB (38GB min) Lower bpw quants and 8-bit cache size can reduce memory requirements / expand context, but if you're sticking to mainstream L1/L2 models, just go by that.
  • coder543 3y ago
    For 128k context (even with a 7B model), I don't think 8GB is nearly enough. I've heard there might be tricks to get it under 24GB... but I haven't personally seen it under about 28GB, IIRC. Definitely not heard of it being possible with 8GB.
• kiviuq 3y ago
  there's a calculator / estimator somewhere on Hugging Face that can give you some approximate numbers based on the model you select.
• brucethemoose2 3y ago
  With exllama, the number is "flexible" because it supports completely arbitrary quantization levels. 7B fits comfortably in 6GB. 70B will squeeze, into 24GB, albeit with significant (but not catastrophic) quality loss.