4 ms·
Agreed. ViTs are better if you're looking to go multimodal or use attention-specific mechanisms such as cross-attention. If not, there's evidence out there that
by fzliu 2y ago
Agreed. ViTs are better if you're looking to go multimodal or use attention-specific mechanisms such as cross-attention. If not, there's evidence out there that ViTs are not better than convnets for small networks and at scale (https://frankzliu.com/blog/vision-transformers-are-overrated https://frankzliu.com/blog/vision-transformers-are-overrated).
- abrichr 2y agoViTs also have proven to be more effective for zero-shot generalization tasks due to their ability to capture global context and relationships in the input data, which CNNs struggle with. https://arxiv.org/abs/2304.02643 https://arxiv.org/abs/2304.02643