3 ms·
You can't fit the model into 4090 without quantization, its like 64 gigs. For home use, Gemma27B QAT is king. Its almost as good as Deepseek R1
by ActorNightly 1y ago
You can't fit the model into 4090 without quantization, its like 64 gigs.
For home use, Gemma27B QAT is king. Its almost as good as Deepseek R1
- modeless 1y agoThe 20B one fits.
- steinvakt2 1y agoDoes it fit on a 5080 (16gb)?
- jwitthuhn 1y agoHaven't tried myself but it looks like it probably does. The weight files total 13.8 GB which gives you a little left over to hold your context.
- northern-lights 1y agoIt fits on a 5070TI, so should fit on a 5080 as well.
- SirMaster 1y agoYou don't really need it to fit all in VRAM due to the efficient MoE architecture and with llama.cpp The 120B is running at 20 tokens/sec on my 5060Ti 16GB with 64GB of system ram. Now personally I find 20 tokens/sec quite usable, but for some maybe it's not enough.
- dexterlagan 1y agoI have a similar setup but with 32 GB of RAM. Do you partly offload the model to RAM? Do you use LMStudio or other to achieve this? Thanks!
- SirMaster 1y agoYes, LMStudio and it automatically does this.