Yes, self-attention mechanisms are dense associative memories, so it might be possible to replace them in many cases with simpler storage mechanisms. Still, I w

Yes, self-attention mechanisms are dense associative memories, so it might be possible to replace them in many cases with simpler storage mechanisms. Still, I would count the required storage space as part of a model's parameter size -- e.g., a model consisting of 1 trillion values in RAM and 99 trillion values in storage consists of... 100 trillion values.