This mutable working memory is implicit. It comes from the transformer architecture. What the neural network weight are encoding is the dynamic of this implicit

This mutable working memory is implicit. It comes from the transformer architecture. What the neural network weight are encoding is the dynamic of this implicit working memory such that it produces the desired output. If you use something like LSTM instead of transformers, it's easier to see the memory cells. But "transformers are RNNs" even if it's a little harder to see.