4 ms·
Could 16gb vram be enough for the 27b QAT version?
by holografix 1y ago
Could 16gb vram be enough for the 27b QAT version?
- halflings 1y agoThat's what the chart says yes. 14.1GB VRAM usage for the 27B model.
- erichocean 1y agoThat's the VRAM required just to load the model weights. To actually use a model, you need a context window. Realistically, you'll want a 20GB GPU or larger, depending on how many tokens you need.
- oezi 1y agoI didn't realize that the context would require such so much memory. Is this KV caches? It would seem like a big advantage if this memory requirement could be reduced.
- jffry 1y agoWith `ollama run gemma3:27b-it-qat "What is blue"`, GPU memory usage is just a hair over 20GB, so no, probably not without a nerfed context window
- woadwarrior01 1y agoIndeed, the default context length in ollama is a mere 2048 tokens.
- hskalin 1y agoWith ollama you could offload a few layers to cpu if they don't fit in the VRAM. This will cost some performance ofcourse but it's much better than the alternative (everything on cpu)
- senko 1y agoI'm doing that with a 12GB card, ollama supports it out of the box. For some reason, it only uses around 7GB of VRAM, probably due to how the layers are scheduled, maybe I could tweak something there, but didn't bother just for testing. Obviously, perf depends on CPU, GPU and RAM, but on my machine (3060 + i5-13500) it's around 2 t/s.
- dockerd 1y agoDoes it work on LM Studio? Loading 27b-it-qat taking up more than 22GB on 24GB mac.
- parched99 1y agoI am only able to get the Gemma-3-27b-it-qat-Q4_0.gguf (15.6GB) to run with a 100 token context size on a 5070 ti (16GB) using llamacpp. Prompt Tokens: 10 Time: 229.089 ms Speed: 43.7 t/s Generation Tokens: 41 Time: 959.412 ms Speed: 42.7 t/s
- floridianfisher 1y agoTry one of the smaller versions. 27b is too big for your gpu
- parched99 1y agoI'm aware. I was addressing the question being asked.
- tbocek 1y agoThis is probably due to this: https://github.com/ggml-org/llama.cpp/issues/12637 https://github.com/ggml-org/llama.cpp/issues/12637. This GitHub issue is about interleaved sliding window attention (iSWA) not available in llama.cpp for Gemma 3. This could reduce the memory requirements a lot. They mentioned for a certain scenario, going from 62GB to 10GB.
- nolist_policy 1y agoOllama supports iSWA.
- parched99 1y agoResolving that issue, would help reduce (not eliminate) the size of the context. The model will still only just barely fit in 16 GB, which is what the parent comment asked. Best to have two or more low-end, 16GB GPUs for a total of 32GB VRAM to run most of the better local models.
- idonotknowwhy 1y agoI didn't realise the 5070 is slower than the 3090. Thanks. If you want a bit more context, try -ctv q8 -ctk q8 (from memory so look it up) to quant the kv cache. Also an imatrix gguf like iq4xs might be smaller with better quality
- abawany 1y agoI tried the 27b-iat model on a 4090m with 16gb vram with mostly default args via llama.cpp and it didn't fit - used up the vram and tried to use about 2gb of system ram: performance in this setup was < 5 tps.