4 ms·
An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets
by ThunderSizzle 3mo ago
An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.
- syntaxing 3mo agoWhich quants? I get these speed (only 20-30 TPS) at Q4_K_M with MTP for 27B on my framework desktop. I draw sub 130W for the whole machine.
- acchow 3mo agoThat's about right. Half the TPS with half the power. GLM-5.2 will be much more demanding tho
- mhjkl 3mo agoIf you’re on AMD, use llamacpp with the vulkan backend. For the 35B I get around 115 tps without mtp and 150-170tps with it. Also the pcie slot you use affects it
- ThunderSizzle 3mo agoI have issues with the vulkan backend. It seems to cause over all GUI sluggishness, but maybe that was actually Firefox. I'll give it another go.