3 ms·
this is explicitly why qwen opted to use recurrent layers as the middle layers of their hybrid model.
by FuckButtons 3mo ago
this is explicitly why qwen opted to use recurrent layers as the middle layers of their hybrid model.
- HarHarVeryFunny 3mo agoAFAIK Alibaba's Qwen3-Next architecture doesn't do anything different specifically for the middle layers - it uses a 3:1 mix of recurrent and regular attention blocks throughout the full depth of the transformer. The motivation for this mostly recurrent hybrid attention is to efficiently support long context lengths.