3 ms·
For more context to people, we have the universal approximation theorem for neural nets that basically says if a network is wide enough it can approximate anyth
by godelski 5y ago
For more context to people, we have the universal approximation theorem for neural nets that basically says if a network is wide enough it can approximate anything (with at least 2 layers). So a lot of stuff was really wide. Then VGG[0] came out and showed that deep networks were very effective (along with other papers, things happen in unison. Leibniz and Newton). Then you get ResNets[1] with skip connections and move forward to today. Today we've started looking more at what networks are doing and where their biases lie. This is because we're running into some roadblocks with CNNs vs Transformers. They have different inductive biases. Vision transformers still aren't defeating CNNs, but they are close and it is clear they learn different things. So we're seeing more papers doing these types of analyses. ML will likely never be fully interpretable, but we're getting better at understanding. This is good because a lot of times picking your model and network architecture is more art than science (especially when choosing hyper parameters).
[0] https://arxiv.org/abs/1409.1556 https://arxiv.org/abs/1409.1556
[1] https://arxiv.org/abs/1512.03385 https://arxiv.org/abs/1512.03385
- joe_the_user 5y agoDeep networks have also been shown to be universal, just FYI,
- godelski 5y agoOh yes, sorry if it gets interpreted otherwise. It is too late to edit for that clarity but thanks for the note.
- dimatura 5y agoI would say AlexNet [1], rather than VGG, was the landmark paper that got the computer vision community to pay attention to deep learning, specifically by winning the 2012 imagenet competition by a large margin. Not that there weren't successes before (specifically, deep nets had also been getting traction in speech processing), and of course deep learning itself is much older than alexnet. But I think most people, specially in vision, would say the 2012 imagenet competition was the watershed moment for DL. By current standards it's not very deep, but at the time it was definitely "deeper"than the popular models at the time (which were mostly not neural networks). VGG is also super influential of course -- it reinforced the trend towards ever deeper networks, which ResNet also took to another level. [1] https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf https://papers.nips.cc/paper/4824-imagenet-classification-wi...
- godelski 5y agoAlexNet really highlighted the importance of convolutions (especially with respect to vision). I'm not disagreeing with you, but there's of course nuance. I thought VGG better highlighted the depth vs breadth argument whereas AlexNet was more CNN vs Linear.