3 ms·
It depends on your usecase and the size of your ram is not the only driver. I think the primary performance drivers (without sacrificing precision) right now ar
by silentbob7 2mo ago
It depends on your usecase and the size of your ram is not the only driver. I think the primary performance drivers (without sacrificing precision) right now are QAT, MTP/DFlash, MoE and Hybrid approaches to avoid full attention in parts by replacing with linear / sparse attention.
So Gemma4 26B A4B QAT+MTP (from unsloth) would be a good pick atm. I'd love to see some smaller models with all that features.