5 ms·
Is there any good reason why a fully-connected network needs more than one hidden layer? Theoretically, any non-linear function could be mapped into a fqn with
by la_fayette 6y ago
Is there any good reason why a fully-connected network needs more than one hidden layer? Theoretically, any non-linear function could be mapped into a fqn with only one hidden layer. Does deep has anything to do with fqns or only with cnns?
- fantod 6y agoIIRC the number of hidden units required scales exponentially in the desired approximation error.
- bobbylarrybobby 6y agoIf you have only a single layer, aren’t you limiting the functions you can approximate to linear combinations of your activation function on the inputs? With deep networks you can take functions of linear combinations of functions of linear combinations of...
- webmaven 6y ago> Is there any good reason why a fully-connected network needs more than one hidden layer? A sufficiently (infinitely?) wide shallow fcnn can work in theory, but is basically impossible to train[0]. It might be feasible to transform or 'compile' a trained deep network into a shallow & wide one, but I'm not sure there would be any benefits, absent sufficiently wide parallel hardware[1] [0] Then again, deep networks were impossible to train for quite a while too, due to the exploding gradient problem. [1] Although, the Cerebras Wafer-Scale chip does exist. Hmm.