3 ms·
My understanding is that as a general rule: ( Q / 8 ) * B = GB RAM, where Q is the Quant level and B is the model size. So a Q4 7B model is ( 4 / 8 ) * 7 = 3.
by CobaltFire 2y ago
My understanding is that as a general rule:
( Q / 8 ) * B = GB RAM, where Q is the Quant level and B is the model size.
So a Q4 7B model is ( 4 / 8 ) * 7 = 3.5GB RAM (or VRAM).
A non-Q model is 16, so 2 * B.
I believe this is before context, which adds a bit.
- SushiHippie 2y agoDefinitely does not seem too far off: https://ollama.com/library/llama3/tags https://ollama.com/library/llama3/tags Llama3 8B fp16 is 16GB, q8 is 8.5GB and q4 is between 4.3 and 4.9GB. Llama3 70B fp16 is 141GB, q8 75GB and q4 is between 40 and 43GB.
- tmaly 2y agobased on your formulas, I am definitely going to need a larger card.