2 ms·
Mainline consumer cards are 16GB, so everyone wants models they can run on their $400 GPU.
by WarmWash 7mo ago
Mainline consumer cards are 16GB, so everyone wants models they can run on their $400 GPU.
- NekkoDroid 7mo agoYea, I've been waiting a while for a model that is ~12-13GB so there is still a bit of extra headroom for all the different things running on the system that for some reason eat VRAM.
- vparseval 7mo agoI found that you can run models locally pretty well that exceed your VRAM by a bit. At least ollama will hand excess off to your system RAM. Maybe performance suffers but I've never actually seen it crap out and I can wait a few minutes for a response.