30 ms·

So about these deep neural networks: I seem to remember from some introductory AI class many years ago that a multilayer neural network can't do anything a sing

by lutorm 12y ago

So about these deep neural networks: I seem to remember from some introductory AI class many years ago that a multilayer neural network can't do anything a single-layer one can't. So what's the deal here then? Is it because the algorithms for learning can't be reduced to an equivalent single-layer network?

thisisdave 12y ago
The argument is that single-layer networks would need to be impractically wide to have the same representational power as a narrower network with multiple layers. I haven't looked into the math myself, but apparently there are situations where the increase in representational power for networks of a given size can be increased exponentially by moving nodes into higher layers.
dimatura 12y ago
In theory, at least, a deep architecture can learn the same things as a shallow one, but with less nodes. See http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/239 http://www.iro.umontreal.ca/~lisa/publications2/index.php/pu...
Houshalter 12y ago
There isn't anything a large number of if-then statements can't do either. Why do programmers use nested if-then statements or states?
- cma 12y ago
  For some GPUS, all code runs, no nested its, just flags to turn unused paths into no-ops. All I'm trying to say is even in your example, the optimal thing we could come up with depended on the problem at hand (graphics vs general purpose). Newer GPUs are different, but still, the ridicule of his question isn't appropriate, reasoning by analogy is error prone, and the real answer to his question is probably a lot more rigorous.
  - Houshalter 12y ago
    This is because GPUs are parallel and can't do nested ifs, not because representing code that way is efficient. Even GPUs do sequential operations. You could never program the vast majority of algorithms like "if the input is exactly 100011110101... then output 100011001...". If that seems silly, that is exactly how the proof that single layer NNs are perfectly general works. It proves that you can represent any series of if then statements like that in an NN. And people who don't understand the proof are mislead into thinking single layer NNs are just as good as deep NNs.

by lutorm 12y ago

thisisdave 12y ago

The argument is that single-layer networks would need to be impractically wide to have the same representational power as a narrower network with multiple layers. I haven't looked into the math myself, but apparently there are situations where the increase in representational power for networks of a given size can be increased exponentially by moving nodes into higher layers.

dimatura 12y ago

In theory, at least, a deep architecture can learn the same things as a shallow one, but with less nodes. See http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/239 http://www.iro.umontreal.ca/~lisa/publications2/index.php/pu...

Houshalter 12y ago

There isn't anything a large number of if-then statements can't do either. Why do programmers use nested if-then statements or states?

cma 12y ago
For some GPUS, all code runs, no nested its, just flags to turn unused paths into no-ops. All I'm trying to say is even in your example, the optimal thing we could come up with depended on the problem at hand (graphics vs general purpose). Newer GPUs are different, but still, the ridicule of his question isn't appropriate, reasoning by analogy is error prone, and the real answer to his question is probably a lot more rigorous.
- Houshalter 12y ago
  This is because GPUs are parallel and can't do nested ifs, not because representing code that way is efficient. Even GPUs do sequential operations. You could never program the vast majority of algorithms like "if the input is exactly 100011110101... then output 100011001...". If that seems silly, that is exactly how the proof that single layer NNs are perfectly general works. It proves that you can represent any series of if then statements like that in an NN. And people who don't understand the proof are mislead into thinking single layer NNs are just as good as deep NNs.