3 ms·
Linear attention will cause it to forget details and make it less rich than alternatives using proper quadratic attention. It only competes with the best transf
by treprinum 3y ago
Linear attention will cause it to forget details and make it less rich than alternatives using proper quadratic attention. It only competes with the best transformers in benchmarks when the number of parameters is about the same anyway whereas one would expect a drastic decrease in parameters when using linear attention.
- sebzim4500 3y agoThat could still be a good trade off. It's fine for the feedforward blocks to be a bit slower (due to higher model dimension) if the 'attention' blocks are much faster (due to better complexity).