3 ms·
You need to pick the correct model size and quantization for the amount of GPU RAM you have. For any given model don’t download the default file, instead go to
by M4v3R 3y ago
You need to pick the correct model size and quantization for the amount of GPU RAM you have. For any given model don’t download the default file, instead go to Tags section on Ollama’s page and pick a quantization whose size in GB is at most 2/3rd of your available RAM, and it should work. For example in your case Mistral-7B q4_0 and even q8_0 should work perfectly.
- swyx 3y agowhats the intuition for 2/3 of RAM?
- avereveard 3y agoyou need some leftover for holding the context
- M4v3R 3y agoBecause there’s always some overhead during inference plus you don’t want to fill all your available RAM because you risk swapping to disk which will make everything slow to a crawl.
- swyx 3y agoso why is the overhead a 1/3 ratio instead of a constant amount? just testing the scaling assumption