4 ms·
No, ALL machine learning architectures are convolutions https://grlearning.github.io/papers/11.pdf https://grlearning.github.io/papers/11.pdf
by adamnemecek 3y ago
No, ALL machine learning architectures are convolutions https://grlearning.github.io/papers/11.pdf https://grlearning.github.io/papers/11.pdf
- Legend2440 3y agoReading the paper, they're saying convolutions are powerful enough to express any possible architecture. But that's just computational universality - this does not mean they are convolutions.
- atq2119 3y agoEvery matrix multiply can be viewed as a 1x1 convolution, just like every convolution can be viewed as an (implicit, for larger than 1x1 kernels) matrix multiply. I'm not sure this is particularly enlightening, but it's probably one small step of understanding that is required to truly "get" the underlying math. Edit: Should have said that every matrix multiply against weights is a convolution. The matrix multiply in the standard attention block (query times keys) really doesn't make sense to be seen as a convolution.
- adamnemecek 3y agoIntegral transform is the right abstraction level. Matmul is an integral transform.