4 ms·

Thanks! Yes, great suggestion. One could also use a similar trick to learn the number of layers, by regularizing unused layers to be close to the identity fun

Thanks! Yes, great suggestion. One could also use a similar trick to learn the number of layers, by regularizing unused layers to be close to the identity function. One thing that we wanted to try but didn't have time was to learn a generalization of convolution. Since convnets are a special case of fully-connected nets with some weights tied and some set to zero, we were thinking of ways to parameterize this weight tying that included both fully-connected and convnets as a special case.

taliesinb 11y ago
> One thing that we wanted to try but didn't have time was to learn a generalization of convolution. Since convnets are a special case of fully-connected nets with some weights tied and some set to zero, we were thinking of ways to parameterize this weight tying that included both fully-connected and convnets as a special case. I see, that's cool. So you could define a 'symmetry operator' the defines a regularization term as e.g. k * sum_{ij}[(w_{ij} - w_{i+i',j+j'})^2] (modulo my hand-wavy tensor indexing). Then the dk tells you how much a layer wants to be a convolution and di', dj' tells you what convolution kernel size it wants to be. (edit: made less badly articulated)
- duvenaud 11y ago
  Exactly. The problem is that it's hard to find a good balance between having too many hyperparameters, versus basically telling it the answer beforehand. In particular, it was hard to figure out how to let it learn the stride of the convolution, but perhaps we could have done without it. The other problem is that to get a reasonably-sized convnet by pruning and tying a fully-connected net, you need to start with a huge fully-connected net, which slows things down.
  - taliesinb 11y ago
    > In particular, it was hard to figure out how to let it learn the stride of the convolution, but perhaps we could have done without it. Yeah, it seems impossible because changing the stride changes the size of the output tensor completely.

taliesinb 11y ago

> One thing that we wanted to try but didn't have time was to learn a generalization of convolution. Since convnets are a special case of fully-connected nets with some weights tied and some set to zero, we were thinking of ways to parameterize this weight tying that included both fully-connected and convnets as a special case. I see, that's cool. So you could define a 'symmetry operator' the defines a regularization term as e.g. k * sum_{ij}[(w_{ij} - w_{i+i',j+j'})^2] (modulo my hand-wavy tensor indexing). Then the dk tells you how much a layer wants to be a convolution and di', dj' tells you what convolution kernel size it wants to be. (edit: made less badly articulated)

duvenaud 11y ago
Exactly. The problem is that it's hard to find a good balance between having too many hyperparameters, versus basically telling it the answer beforehand. In particular, it was hard to figure out how to let it learn the stride of the convolution, but perhaps we could have done without it. The other problem is that to get a reasonably-sized convnet by pruning and tying a fully-connected net, you need to start with a huge fully-connected net, which slows things down.
- taliesinb 11y ago
  > In particular, it was hard to figure out how to let it learn the stride of the convolution, but perhaps we could have done without it. Yeah, it seems impossible because changing the stride changes the size of the output tensor completely.