I don't remember the contents of that paper, but I can give you some context based on my knowledge of "traditional" theoretical computer science. In theoretica

I don't remember the contents of that paper, but I can give you some context based on my knowledge of "traditional" theoretical computer science. In theoretical CS you have state machines, pushdown automatons and turing machines. What may surprise you is that the difference between those three does not lie in the way the algorithm for them is represented. They all use state transition diagrams! Pushdown automatons are more powerful than state machines, because they have a stack onto which they can push new data, peek at the top of the stack or pop data off the stack. Now here is the kicker! How do you get a turing machine? You take a pushdown automaton and remove the restriction that you can only push, peek or pop at the head! You can now move the head pointing at the stack, your stack has turned into a tape! The key difference lies in the datastructure that the transition diagram is manipulating, not the algorithm itself! The attention mechanism of the transformer architecture is analogous to a tape that can only be written to once per blank and in theory that alone is enough to emulate a full Turing machine with read, write, rewrite and delete semantics.