3 ms·

Has anyone tried searching for new basic operations like convolution or pooling? We've been using these methods for years, and I doubt the first major developm

Has anyone tried searching for new basic operations like convolution or pooling? We've been using these methods for years, and I doubt the first major development in NN vision (convolution) is the most optimal method possible. Consider the extreme case of searching over all (differentiable) mathematical operations to see if something really novel can be discovered. How feasible would this be?

jellyksong 5y ago
This is basically what the field of neural architecture search tries to do. Here’s a good (somewhat technical) introduction: https://lilianweng.github.io/lil-log/2020/08/06/neural-architecture-search.html https://lilianweng.github.io/lil-log/2020/08/06/neural-archi...
- IdiocyInAction 5y ago
  I think NAS is a bit higher level than what the OP had in mind - NAS isn't usually used to search for fundamental operations like self-attention or convolution. But I guess you could probably adapt it quite easily.
  - jellyksong 5y ago
    I believe some algorithms, like AutoML-Zero, search on the level of mathematical operations.
IdiocyInAction 5y ago
Multi-head self-attention seems to be the new trendy architectural primitive. I don't know how feasible it would be - I guess you could take a set of base operations (matrix multiplication, softmax, etc.) and randomly generate feature transformations and check if any of them yield good features (stick a linear readout at the end of it and test the performance on some downstream tasks). That would be an unguided search - I guess you could try something like GA or something. Also, it uses neural network training as an inner loop step, so it would probably be to expensive. Better would be if you could get the gradient w.r.t. to the tentative operation somehow. Problem is that training NNs is nontrivial and you might need things like BatchNorm and residual connections to make things stable, so you'd somehow have to search for good architectures for each operation as well.
gspr 5y ago
At least there is work on greatly generalizing convolutions. They're much more broadly applicable (in neural networks) to very differently structured data than they appear to be in their standard form. (The "in neural networks" qualifier is there because quite a bit of this has been understood about the mathematical operation convolution for a long long time). Some recent developments: * https://arxiv.org/abs/2010.03633 https://arxiv.org/abs/2010.03633 (disclosure: I'm one of the authors) * https://arxiv.org/abs/2012.06333 https://arxiv.org/abs/2012.06333
xiaodai 5y ago
group CNN are drop in replacement of convolution layers.

jellyksong 5y ago

This is basically what the field of neural architecture search tries to do. Here’s a good (somewhat technical) introduction: https://lilianweng.github.io/lil-log/2020/08/06/neural-architecture-search.html https://lilianweng.github.io/lil-log/2020/08/06/neural-archi...

IdiocyInAction 5y ago
I think NAS is a bit higher level than what the OP had in mind - NAS isn't usually used to search for fundamental operations like self-attention or convolution. But I guess you could probably adapt it quite easily.
- jellyksong 5y ago
  I believe some algorithms, like AutoML-Zero, search on the level of mathematical operations.

IdiocyInAction 5y ago

Multi-head self-attention seems to be the new trendy architectural primitive. I don't know how feasible it would be - I guess you could take a set of base operations (matrix multiplication, softmax, etc.) and randomly generate feature transformations and check if any of them yield good features (stick a linear readout at the end of it and test the performance on some downstream tasks). That would be an unguided search - I guess you could try something like GA or something. Also, it uses neural network training as an inner loop step, so it would probably be to expensive. Better would be if you could get the gradient w.r.t. to the tentative operation somehow. Problem is that training NNs is nontrivial and you might need things like BatchNorm and residual connections to make things stable, so you'd somehow have to search for good architectures for each operation as well.

gspr 5y ago

At least there is work on greatly generalizing convolutions. They're much more broadly applicable (in neural networks) to very differently structured data than they appear to be in their standard form. (The "in neural networks" qualifier is there because quite a bit of this has been understood about the mathematical operation convolution for a long long time). Some recent developments: * https://arxiv.org/abs/2010.03633 https://arxiv.org/abs/2010.03633 (disclosure: I'm one of the authors) * https://arxiv.org/abs/2012.06333 https://arxiv.org/abs/2012.06333

xiaodai 5y ago

group CNN are drop in replacement of convolution layers.