4 ms·
Just checking what you are saying in your last paragraph: do you mean to say that shallow NN's don't overfit?
by sgt101 12y ago
Just checking what you are saying in your last paragraph: do you mean to say that shallow NN's don't overfit?
- Houshalter 12y agoIf I understand correctly, they generated artificial data from the NN they were mimicking.
- wodenokoto 12y agoNo, shallow NN definitely overfits in normal circumstances. However, in this case they are building a shallow NN that is not learning a normal function, but is being trained to emulate a deep network of equivalent amount of parameters. Parents claim is that in that case, we don't have to worry about over fitting, since we want every result of target network to be as closely emulated as possible in the shallow network.