3 ms·

It's interesting that it's the middle layers of the Transformer that are affected most by RL post-training, but it perhaps makes some intuitive sense given that

It's interesting that it's the middle layers of the Transformer that are affected most by RL post-training, but it perhaps makes some intuitive sense given that RL is being used to shape high level planning-type direction of the output. It seems that the input layers to a Transformer are necessarily going to be doing the most low level work of syntax -> semantic augmentation starting with things like tagging parts of speech etc. Similarly the output layers are by necessity going to be concerned with mapping high level representations back into surface level word sequence form. This leaves the middle layers to do the work of first recognizing deep enough patterns to support good quality prediction, then do the high level predication itself which is what RL is typically going to be trying to shape.

FuckButtons 4mo ago
this is explicitly why qwen opted to use recurrent layers as the middle layers of their hybrid model.
- HarHarVeryFunny 4mo ago
  AFAIK Alibaba's Qwen3-Next architecture doesn't do anything different specifically for the middle layers - it uses a 3:1 mix of recurrent and regular attention blocks throughout the full depth of the transformer. The motivation for this mostly recurrent hybrid attention is to efficiently support long context lengths.
godelski 4mo ago
> the middle layers of the Transformer that are affected most by RL post-training This is where you should expect most change in models. The beginning layers need to embed while the later layers will reform the result to the final conditions. The middle is what does all the untangling.