3 ms·
I’m not an LLM guy… how large on disk is a 7B LLM? (Minstrel is specific is needed, but a range would be great)
by SV_BubbleTime 3y ago
I’m not an LLM guy… how large on disk is a 7B LLM? (Minstrel is specific is needed, but a range would be great)
- Feorn 3y agoA completely unquantized fp16 model weight 7B LLM is about 15GB on a disk. You need closer to 24GB of memory for inference with a decently sized context. Quantization is black magic of the software variety that seems to be able to significantly reduce that without a commensurate loss in quality, though the results are a little subjective. Some well reviewed quantizations of 7B models can get them below 9GB.