4 ms·
~18.6 GiB, according to nvtop. ollama 0.6.6 invoked with: # server OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve # client o
by genpfault 1y ago
~18.6 GiB, according to nvtop.
ollama 0.6.6 invoked with:
# server
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
# client
ollama run --verbose qwen3:30b-a3b
~19.8 GiB with:
/set parameter num_ctx 32768
- tgtweak 1y agoVery nice, should run nicely on a 3090 as well. TY for this. update: wow, it's quite fast - 70-80t/s on LM Studio with a few other applications using GPU.