3 ms·
For "full" context, typically: 7B: 8GB (4GB min for model only) 13B: 12GB (8GB min) 34B: 28GB (18GB min) 70B: 60GB (38GB min) Lower bpw quants
by washadjeffmad 3y ago
For "full" context, typically:
7B: 8GB (4GB min for model only)
13B: 12GB (8GB min)
34B: 28GB (18GB min)
70B: 60GB (38GB min)
Lower bpw quants and 8-bit cache size can reduce memory requirements / expand context, but if you're sticking to mainstream L1/L2 models, just go by that.
- coder543 3y agoFor 128k context (even with a 7B model), I don't think 8GB is nearly enough. I've heard there might be tricks to get it under 24GB... but I haven't personally seen it under about 28GB, IIRC. Definitely not heard of it being possible with 8GB.