2 ms·
The goal here is not to replace transformers but combine them with RNN so you get both good short-term memory (self-attention) and much improved long-term memor
by tshadley 1y ago
The goal here is not to replace transformers but combine them with RNN so you get both good short-term memory (self-attention) and much improved long-term memory (ATLAS recurrent memory).
"Empirically, our models—OmegaNet, Atlas, DeepTransformers,
and Dot—achieve consistent improvements over Transformers and recent RNN variants across diverse benchmarks."