"Figure 2: Recurrent memory mechanism. Memory is passed to Transformer along input sequence embeddings, and memory output is passed to the next segment. During

"Figure 2: Recurrent memory mechanism. Memory is passed to Transformer along input sequence embeddings, and memory output is passed to the next segment. During training gradients flow from the current segment through memory to the previous segment." The whole point of the 2017 paper "Attention is all you need" was to show that the recurrent structure previously used (with attention mechanisms) wasn't needed. It's pretty cool if it does turn out that both mechanisms are needed and powerful together. After all, the brain doesn't linearly process the entire book you're reading from page 1 to where you are, for every new word you read, it stashes the important parts into short/long term memory along the way. Still, passing gradients back through several stages like that is typically a tricky thing to do in practice.