3 ms·
Just this morning I tweaked my single 3090 setup too: OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=180000 and that fits in 2
by hughw 4mo ago
Just this morning I tweaked my single 3090 setup too:
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_CONTEXT_LENGTH=180000
and that fits in 23GB.
[edited for format]
- MaKey 4mo agoFriends don't let friends use Ollama: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/
- hasteg 4mo agoJust a heads up (you may already know this) -- In the local LLM community there's a pretty strong disdain for Ollama specifically due to numerous reasons provided in this blog post: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/