I recommend checking out this paper: https://arxiv.org/pdf/2310.16764.pdf https://arxiv.org/pdf/2310.16764.pdf. From the concluding section: "Although the succ

I recommend checking out this paper: https://arxiv.org/pdf/2310.16764.pdf https://arxiv.org/pdf/2310.16764.pdf. From the concluding section: "Although the success of ViTs in computer vision is extremely impressive, in our view there is no strong evidence to suggest that pre-trained ViTs outperform pre-trained ConvNets when evaluated fairly." Neural networks can be unpredictable, and there's evidence that questions how important transformers' lack of inductive bias (at scale) really is.