The most important bits about what they do to make this happen: > In addition to the causal self-attention integral to transformers, we also allow each query t

The most important bits about what they do to make this happen: > In addition to the causal self-attention integral to transformers, we also allow each query token to attend to a fixed number of “external memories”. These memories are stored in a non-differentiable cache. The choice of which memories to attend to is made using cosine similarity within each decoder layer and attention head. [...] > We create our external memories (at each layer) by passing those external contexts through our model, just like inference. Then we save the internal representations the model generated, and attend to them later. What an extremely clever approach! If, in a chatbot setting, you update the external memory cache during inference, this means the thing immediately retains memory of the discussion. Maybe this is an alternative (quicker? more exact?) to LoRA finetuning, to give a foundational model some specific personality and experiental history?