4 ms·
> It can run 80-100t/s on a laptop That is a lot, what is your laptop hardware? One issue I have with Gemma is that they seem to use old architectures that re
by tarruda 1mo ago
> It can run 80-100t/s on a laptop
That is a lot, what is your laptop hardware?
One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled.
Qwen 3.5+ is much better in that regard with its super efficient context. Even on Macs, speeds take degrade much more slowly.
- dofm 1mo ago> One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled. Yes, this is something I hope they will change. Gemma 4 31B is much slower on pre-Blackwell GPUs as a result, which is a bit of a shame for local model experimentation.
- rao-v 1mo agoEven Muse Glimmer (as did GPT-OSS I think) does ~4 sliding window attention layers + 1 full attention layer (like Gemma 4). I’m assuming both labs have good reason to think that gated delta nets are not optimal. Of course it’s possible the labs just stick with the optimal architecture for large models and GDN is best for smaller models.
- dofm 1mo agoThanks for the reply. Sooo much I have to learn.