3 ms·
https://arxiv.org/abs/1904.09925 https://arxiv.org/abs/1904.09925
by p1esk 8y ago
https://arxiv.org/abs/1904.09925 https://arxiv.org/abs/1904.09925
- nl 8y agoNice. I guess I'm 8 days behind on the SOTA... But I'd note that it is build on top of a CNN base (ResNet or RetinaNet) and that the Attention-only system performed slightly worse than the one including the CNN layers. Also, this isn't really a Transformer architecture, even though it uses Attention. But maybe this is too much nitpicking? I agree that Attention is a useful primitive - my point is that the Transformer architecture is too specific. (Also, this is a really nice paper in that it lays out the hyperparameters and training schedules they used. And that Appendix is amazing!)