2 ms·
Nice. I guess I'm 8 days behind on the SOTA... But I'd note that it is build on top of a CNN base (ResNet or RetinaNet) and that the Attention-only system perf
by nl 8y ago
Nice. I guess I'm 8 days behind on the SOTA...
But I'd note that it is build on top of a CNN base (ResNet or RetinaNet) and that the Attention-only system performed slightly worse than the one including the CNN layers.
Also, this isn't really a Transformer architecture, even though it uses Attention.
But maybe this is too much nitpicking? I agree that Attention is a useful primitive - my point is that the Transformer architecture is too specific.
(Also, this is a really nice paper in that it lays out the hyperparameters and training schedules they used. And that Appendix is amazing!)