5 ms·
I'm not an expert, but I became aware of this paper recently: "A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP" https://arxiv.o
by astroalex 5y ago
I'm not an expert, but I became aware of this paper recently:
"A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP"
https://arxiv.org/pdf/2108.13002.pdf https://arxiv.org/pdf/2108.13002.pdf
I can't understand the technical jargon, but my interpretation is that MLP turns out to not be as good as CNN / Transformer. Maybe someone with more expertise can weigh in!
- fxtentacle 5y agoMLPs discard more spatial information. That is great for "Do you see a cat?" but bad for "Where is the cat?". So which one is better depends on your use case.
- omegalulw 5y agoMLP don't "discard" spatial information. Rather, spatial invariance is a very useful inductive bias for learning that MLPs don't have. If you had a perfect optimizer MLP would outperform CNNs (assuming hard enough tasks) but you don't have a perfect optimizer.
- ThereIsNoWorry 5y agoMLP is basically the vanilla neural network. The thing everyone sees first when getting taught about deep learning or similar. It turned out that (usually) MLPs are not very efficient to process matrix structures with it (e.g. 2d matrices like images). So, computer vision invented convolutional neural networks specifically to make image data highly efficient to process for neural networks. They outperformed MLPs in basically all aspects across the board. Transformers / Attention is a relatively new invention initially made to solve NLP problems more efficiently; but as it turns out, they work great on images as well. TLDR - we've come full circle.