3 ms·
Great result :) Looks like the attention layers aren't as important as everyone thought because using a similarly sized feed-forward layer works almost as well
by fxtentacle 4y ago
Great result :)
Looks like the attention layers aren't as important as everyone thought because using a similarly sized feed-forward layer works almost as well (74% vs 77% top1 accuracy)
- BobbyJo 4y agoWouldn't attention networks use less computation for the same number of weights? Feed forward networks have higher connectivity no? If a network is the same size, less computationally demanding, and gives you 3% improvement, it seems extremely worthwhile, especially given the (mostly) diminishing returns of just adding more weights/layers.
- jeffreyrogers 4y agoIt is less general though, so this is interesting because it shows you can learn a lot without adding very much structure.
- Teodolfo 4y agoGoing from 25% top-one error to 22% top-one error is a massive jump on ImageNet and very meaningful in a lot of applications. That said, there is no reason to believe attention-based models are the only way forward on image classification. The humble ResNet-50 can get near 22% top-1 error when trained properly. Whether we need attention or not is a more interesting question on seq2seq models on text data.
- amelius 4y agoWhat top-1 error do humans get on this problem?