It is MoE. It needs to engage multiple experts when the problem is complex or unclear. So you naturally see more of those simply as a primitive it learns to use

It is MoE. It needs to engage multiple experts when the problem is complex or unclear. So you naturally see more of those simply as a primitive it learns to use to page in more diverse set of weights. Remember that each token is just 6 experts out of 256. So it literally needs to tell its router that it needs a different set the next time. And this memory control primitive leaks into the reasoning chain, because it has no other channel for it available and we do not know how to train any other channel. On the flip side, it tends to converge quickly, roughly proportional to the actual difficulty / clarity of the task.