4 ms·
What's the best speed people have gotten on 4090s?
by modeless 1y ago
What's the best speed people have gotten on 4090s?
- ActorNightly 1y agoYou can't fit the model into 4090 without quantization, its like 64 gigs. For home use, Gemma27B QAT is king. Its almost as good as Deepseek R1
- modeless 1y agoThe 20B one fits.
- steinvakt2 1y agoDoes it fit on a 5080 (16gb)?
- jwitthuhn 1y agoHaven't tried myself but it looks like it probably does. The weight files total 13.8 GB which gives you a little left over to hold your context.
- northern-lights 1y agoIt fits on a 5070TI, so should fit on a 5080 as well.
- SirMaster 1y agoYou don't really need it to fit all in VRAM due to the efficient MoE architecture and with llama.cpp The 120B is running at 20 tokens/sec on my 5060Ti 16GB with 64GB of system ram. Now personally I find 20 tokens/sec quite usable, but for some maybe it's not enough.
- dexterlagan 1y agoI have a similar setup but with 32 GB of RAM. Do you partly offload the model to RAM? Do you use LMStudio or other to achieve this? Thanks!
- SirMaster 1y agoYes, LMStudio and it automatically does this.
- asabla 1y agoI'm on a 5090 so it's not apples to apples comparison. But I'm getting ~150t/s for the 20B version using ~16000 context size.
- modeless 1y agoCool, what software?
- asabla 1y agoInitial testing has only been done with ollama. Plan on testing out llama.cpp and vllm when there is enough time
- steinvakt2 1y agoAnd flash attention doesn't work on 5090 yet, right? So currently 4090 is probably faster, or?
- PeterStuer 1y agoI don't think the 4090 has native 4bit support, which will probably have a significant impact.
- diggan 1y ago> And flash attention doesn't work on 5090 yet, right? Flash attention works with GPT-OSS + llama.cpp (tested on 1d72c8418) and other Blackwell card (RTX Pro 6000) so I think it should work on 5090 as well, it's the same architecture after all.