Not an expert, but as I understand it, "attention" is just a learned feature weighting. More important features get higher weights. Less important features get

Not an expert, but as I understand it, "attention" is just a learned feature weighting. More important features get higher weights. Less important features get lower weights. I can see why this would be useful as an explicit step in terms of computation efficiency, e.g. "calculate the most important features and operate on them" However, it seems like such information could easily be learned implicitly by other parts of the model without needing to model attention directly. Which is, I guess, the result in this paper.