8 ms·
Qwen 4B and 9B should be faster and better reasoning than Gemma 4 E4B. Other good options for that much RAM are Gemma 4 12b and 31b. Gemma 4 E4B would be better
by 0xbadcafebee 2mo ago
Qwen 4B and 9B should be faster and better reasoning than Gemma 4 E4B. Other good options for that much RAM are Gemma 4 12b and 31b. Gemma 4 E4B would be better for native audio, but OP is using Whisper for STT so prob doesn't matter
- dofm 2mo agoThe issue is not RAM size. It's memory bandwidth! The 12B and 31B models will be useless on a Pi 5; maybe the 12B can be persuaded to run, but it may not manage more than one token per second. It only manages 17GB/s memory bandwidth (I have seen a suggestion that the 4GB model manages more). But yes — some sort of small reasoning-oriented model (Ornith?) seems a better candidate than Qwen 35B. (Don't get me wrong, I think the 35B model is ace… just seems like at least an unusual choice here)
- petruspennanen 2mo agoHmm could try Ornith - wouldn't mind if answers came a bit faster. Probably hard to reach Qwens intelligence level though.