> Maybe I missed it, but it didn’t seem to me like the authors presented any reasoning about what makes this capability particular to transformers… especially o

> Maybe I missed it, but it didn’t seem to me like the authors presented any reasoning about what makes this capability particular to transformers… especially on the relatively small HMMs, it’s not clear to me that we wouldn’t see the same behavior with some other model performing prediction, but maybe I missed that. I don't believe they are claiming that? There is no reason to think that that would be the case (Transformers are in general grossly overrated in terms of magic pixie dust), the fundamental theory here implies that any predictor achieving high performance is going to have to do this, and in the comments, people point out examples elsewhere like with RNNs where researchers have read off the beliefs from the hidden state and found the same thing: https://www.lesswrong.com/posts/gTZ2SxesbHckJ3CkF/transformers-represent-belief-state-geometry-in-their?commentId=mMvq339M52KBG7q2s https://www.lesswrong.com/posts/gTZ2SxesbHckJ3CkF/transforme...