3 ms·
My intuition for this is that CNNs impose a stronger prior on the function space. We rely on the previous experience of computer vision and even our own retinas
by huevosabio 3y ago
My intuition for this is that CNNs impose a stronger prior on the function space. We rely on the previous experience of computer vision and even our own retinas to give it a useful prior.
Transformers are more general, they impose a weaker prior and require more data. But by imposing a weaker prior they also are more flexible and adaptable, thus able to take advantage of more data.
- chrishare 3y agoAgree. However I feel that there may still be many priors that are useful in practice, like decomposing the vision task into a hierarchy of sub problems, but as a human - I could be biased.
- huevosabio 3y agoOh absolutely, I think having a prior makes these problems tractable. Otherwise we would just force feed data to fully connected networks!
- Legend2440 3y agoIt would be much better if we had a way to learn good priors -> extract them -> initialize new models with them. We shouldn’t be trying to guess or handcraft them.
- joefourier 3y agoNot all vision transformers have weak priors. Shifted-window transformers and neighborhood attention have priors well suited to images; the latter is showing extremely strong performance in image generation (such as the recent hourglass diffusion which allows pixel space diffusion to be trained orders of magnitudes faster than vanilla attention) and general image classification, and certainly does not need the same dataset size of classical ViTs.