AFAIK Alibaba's Qwen3-Next architecture doesn't do anything different specifically for the middle layers - it uses a 3:1 mix of recurrent and regular attention

AFAIK Alibaba's Qwen3-Next architecture doesn't do anything different specifically for the middle layers - it uses a 3:1 mix of recurrent and regular attention blocks throughout the full depth of the transformer. The motivation for this mostly recurrent hybrid attention is to efficiently support long context lengths.