This is the "random linear projections as memorization technique" perspective on Transformers. It's not a new idea per se, but nice to see it fleshed out. If y

This is the "random linear projections as memorization technique" perspective on Transformers. It's not a new idea per se, but nice to see it fleshed out. If you dig into this perspective, it does temper any claims of "cognitive behavior" quite strongly, if only because Transformers have such a large capacity for these kinds of "memories".