3 ms·
comments on the X post point out that 8B models is pretty good. What type of VRAM would you need to run 8B locally?
by tmaly 3y ago
comments on the X post point out that 8B models is pretty good.
What type of VRAM would you need to run 8B locally?
- mertbio 3y agoI can run the 8B with my M2 Pro Mac mini with 32GB via Ollama. It takes around one second to get a response from the API.
- zamalek 3y agoI have enough VRAM on my 6900XT (16GB) for 13B quantized models alongside desktop/hyprland usage.
- CobaltFire 3y agoMy understanding is that as a general rule: ( Q / 8 ) * B = GB RAM, where Q is the Quant level and B is the model size. So a Q4 7B model is ( 4 / 8 ) * 7 = 3.5GB RAM (or VRAM). A non-Q model is 16, so 2 * B. I believe this is before context, which adds a bit.
- SushiHippie 3y agoDefinitely does not seem too far off: https://ollama.com/library/llama3/tags https://ollama.com/library/llama3/tags Llama3 8B fp16 is 16GB, q8 is 8.5GB and q4 is between 4.3 and 4.9GB. Llama3 70B fp16 is 141GB, q8 75GB and q4 is between 40 and 43GB.
- tmaly 3y agobased on your formulas, I am definitely going to need a larger card.
- MyFirstSass 3y agoI run 8b on my Macbook air m2 with 16gb's of vram at decent speed of 12 t/s (mlx can probably get 15 t/s). I use the Q5_K_M GGUF, which is >99% the same as the original. I've seen tests that there is close to no divergence with these quantisations, but it rises steeply going lower: https://www.reddit.com/r/LocalLLaMA/comments/1816h1x/how_much_does_quantization_actually_impact_models/ https://www.reddit.com/r/LocalLLaMA/comments/1816h1x/how_muc...
- tosh 3y ago~8GB RAM for a quantized version of 8B