3 ms·
There is a no free lunch theorem in learning, which says that without limiting your hypothesis space, you cannot learn. That means, you need to have some assump
by js8 10y ago
There is a no free lunch theorem in learning, which says that without limiting your hypothesis space, you cannot learn. That means, you need to have some assumptions.
No doubt that DNNs are subject to the same theorem, the only question is, how does their hypothesis space looks like? Does anyone have idea about this? I suspect we don't really know what the DNNs assumptions are.
- yorwba 10y agoThe DNNs assumptions are "The true model can be fit by this network architecture". If you use convolutions, then an additional assumption is translation invariance.
- js8 10y ago> The DNNs assumptions are "The true model can be fit by this network architecture" Yeah, but that's tautological. What I mean can somebody sit down and for a given DNN architecture, write down (at least approximately) the set of functions that it can learn? Or more importantly, what functions it cannot learn? Or at least, how many bits are assumed and how many bits have to be learned? I think that is what bothers people about DNNs. I personally think they are sometimes even inefficient - we are learning them much more parameters (bits) than the actual hypothesis space requires.
- yorwba 10y ago> What I mean can somebody sit down and for a given DNN architecture, write down (at least approximately) the set of functions that it can learn? For a two-layer architecture with ReLU activations and n units in the hidden layer, this is the set of piecewise linear continuous functions with n kinks.
- Klockan 10y agoHow many of those do you need to tell the difference between cats and dogs in random internet pictures?
- ganfortran 10y agoAgree with you. I think what I need to make it more clear, is DNN puts little assumption on data. It usually consumes it as raw pixels, in case of images. The architecture of NN itself is the biggest assumption here, as it is how it will be used to process the data. But they those operators are usually generic, a lot of them just umbrella operators for matrix multiplications + non-linearity in between.
- Houshalter 10y agoI wrote a bit about the theoretical reasons DNNs work here: http://lesswrong.com/lw/m9p/approximating_solomonoff_induction/ http://lesswrong.com/lw/m9p/approximating_solomonoff_inducti... There's some more interesting discussion in the comments there.