The headline is too linkbaity for my taste. The idea is simple: Apply a mixture of multiple affine transformations, e.g., weighted by Softmax attention, instead

The headline is too linkbaity for my taste. The idea is simple: Apply a mixture of multiple affine transformations, e.g., weighted by Softmax attention, instead of just a single affine transformation, to renormalize the data. The use of attention likely increases running time both in training and testing. See also Datenstrom's comment: https://news.ycombinator.com/item?id=24528468 https://news.ycombinator.com/item?id=24528468