3 ms·
I wish they were a smidge smaller since 7B LLMs just barely run on a 16GB VRAM GPU (like a T4 server GPU) without quantization shenanigans. Fortunately the lea
by minimaxir 3y ago
I wish they were a smidge smaller since 7B LLMs just barely run on a 16GB VRAM GPU (like a T4 server GPU) without quantization shenanigans.
Fortunately the learnings from finding better 7B models will trickle down, or more will be done with distillation (e.g. Gemini Nano)
- 3abiton 3y agoWhat's wrong with quantization?
- minimaxir 3y agoThere's still a (subjective) generative quality loss, even with recent tricks to minimize it.
- leetharris 3y agoQ8 is generally less than 1% degradation, Q5KM is around 3%. After that is when it starts to really degrade.