The interesting thing though is that attention effectively allows a model to meta-learn based on the current context, so in many ways, it may be thought of as a

The interesting thing though is that attention effectively allows a model to meta-learn based on the current context, so in many ways, it may be thought of as analogous to a brain without long term memory.