5 ms·
All machine learning is convolution.
by adamnemecek 3y ago
All machine learning is convolution.
- Legend2440 3y agoOnly certain types of neural networks use convolutions. It's not universal.
- CamperBob2 3y agoWhat are some examples of types that don't?
- hackerlight 3y agoI think anything that doesn't have an explicit convolution layer? Transformer, MLP, RNN don't automatically have a convolution layer, although for many tasks you can add it in if you want.
- dontwearitout 3y agoThe transformer module is currently dominating ML, and is widely used in text, vision, audio, and video models. It was introduced in 2017 and shows no real signs of being displaced. It has no convolutions. https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture) https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc... http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/
- adamnemecek 3y ago… or does it https://grlearning.github.io/papers/11.pdf https://grlearning.github.io/papers/11.pdf
- dontwearitout 3y agoLots of interesting work out there; time will tell!
- CamperBob2 3y agoIf they use dot products on at least one layer with fully-connected inputs, which they do, along with everything else derived from the basic MLP model, then they're technically performing convolution. Of course, the convolution concept breaks down when nonlinear activation functions are introduced, so I'm not sure the equivalence is really all that profound.
- adamnemecek 3y agoNonlinear activation layers are piecewise linear-ish.
- dontwearitout 3y agoI don't think a dot product between high dimensional vectors is considered a convolution? I'm familiar with convolution between continuous functions, and with kernels in neural networks providing invariance. I'd love to learn more if you have any links that expand on your statement.
- Legend2440 3y agoTransformers, MLPs.
- adamnemecek 3y agoNo, ALL machine learning architectures are convolutions https://grlearning.github.io/papers/11.pdf https://grlearning.github.io/papers/11.pdf
- Legend2440 3y agoReading the paper, they're saying convolutions are powerful enough to express any possible architecture. But that's just computational universality - this does not mean they are convolutions.
- atq2119 3y agoEvery matrix multiply can be viewed as a 1x1 convolution, just like every convolution can be viewed as an (implicit, for larger than 1x1 kernels) matrix multiply. I'm not sure this is particularly enlightening, but it's probably one small step of understanding that is required to truly "get" the underlying math. Edit: Should have said that every matrix multiply against weights is a convolution. The matrix multiply in the standard attention block (query times keys) really doesn't make sense to be seen as a convolution.
- adamnemecek 3y agoIntegral transform is the right abstraction level. Matmul is an integral transform.
- dartos 3y agoAll computers do is add.
- tambourine_man 3y agoAnd conditional jumps
- dartos 3y agoWhat is a jump but an addition to the instruction pointer?
- tambourine_man 3y agoTrue
- c0pium 3y agoRandom forest would like a word.