2 ms·
And recently, a paper titled Attention Is Not All You Need has made the rounds arguing that some of the claims made in the AIAYN paper may have been overstated.
by bjourne 6y ago
And recently, a paper titled Attention Is Not All You Need has made the rounds arguing that some of the claims made in the AIAYN paper may have been overstated. https://arxiv.org/abs/2103.03404 https://arxiv.org/abs/2103.03404
- visarga 6y agoIf you read the title, it only refers to the multi-head-attention part of BERT, excluding the feed forward and skip connections, hence calling it "Pure Attention". > Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth This does not prove the original title was wrong, and this paper is not a counter, but an analysis of a submodule which helps better understanding transformers.