4 ms·
Curious what model you're using that works well on a 16GB card? I very much want to use my 5080 for inference, but everything I've tried so far has either just
by overgard 4mo ago
Curious what model you're using that works well on a 16GB card? I very much want to use my 5080 for inference, but everything I've tried so far has either just not been good enough or painfully slow.
- toyg 4mo agoQwen 3.5-9b-Q4_K_M. I have a 5080 too! For me, the key has been dropping Ollama for Llama.cpp, which is not particularly scary to configure anymore and just skyrocketed performance. I download the models with LM Studio, then run them with llama.cpp.
- brianvoe 4mo agogemma 4 12b
- gargola_ 4mo agoThis can work for some things but I wanted to run hermes with 16gb but 12b is too low and the context was too limited, they recommend 27b