4 ms·

Very interesting and well written paper. They show a deep-neural network (which people thought had more explaining-power-per-parameter) can be "emulated" by a s

by ivan_ah 12y ago

Very interesting and well written paper. They show a deep-neural network (which people thought had more explaining-power-per-parameter) can be "emulated" by a shallow neural network with the same number of parameters. What Ba and Caruana have shown is that deep neural networks---as architecture---are not more powerful. The previous general opinion (at least this is what I thought) was that deep architectures of the form: predictions layer 3 features of features of features layer 2 features of features layer 1 features data layer would be more efficient (i.e. more explaining power per parameter). Nope. Apparently, a shallow neural network of the form predictions layer 1 features (a lot more of them) data layer with an equal number of units in layer 1 as the total number of units in deep network, can obtain the same accuracy. Thus, deep neural networks and shallow neural networks have roughly equivalent explaining-power-per-parameter. A separate result shows deep convolutional neural networks can be simulated by shallow NNs with a simulation overhead of ~10x. Another cool observation they make about "installing" the parameters of ML models. We'll use the following analogy to illustrate the point: ML model <==> hardware ML model parameters <==> software Running program X involves (1) buying/renting a server, (2) compiling X for your architecture, and (3) running X. Of course there is always the alternate step (2') of installing pre-compiled binaries. Similarly, we can think of using a ML model as a three step procedure (1) ML.init() (malloc for Model.parameters), (2) Model.train() (fit parameters to training data), (3) Model.predict(new_datum). The authors point out the existence of a possibility of distributing models parameters as binaries. They demonstrate an alternate step (2') in which they train a ''FancyModel'', then set the parameters of the simple model so it "simulates" the FancyModel: ''Model.parameters = simulate(FancyModel)''. In other words, the model parameters of a ML model need not be learned through the model itself. What I find fascinating is it seems deep architectures don't have any more representational power, but they do have more learning power. Something very interesting is going on...

amalcon 12y ago
The novel part is not that the shallow network can compute the same functions (that is interesting, but that particular result is decades old), but the notion that the shallow network can learn the same functions efficiently. What we already knew: For any given neural network of N layers, there exists a 3-layer neural network computing the same function. What we didn't know: There are (at least reasonably often) ways of training shallow networks that get similar efficiency to the ways of training deep networks given the same functions.
- tiger10guy 12y ago
  Their method for training a shallow network requires that one first train a deep network.
- Houshalter 12y ago
  No, they require training a deep neural network first. Then training the shallow net to mimic the deep one. The claim is that the shallow net can have the same number of parameters as the deep net. It's always been known that a large enough shallow NN can theoretically approximate any function. But that it can do so with few parameters is very surprising. My best guess is that the majority of parameters in deep NNs are unused or redundant. I.e they have a low weight, or don't influence the output very much, or another neuron computes mostly the same function. Many types of regularization like dropOut and weight decay heavily encourage this. Whereas the shallow NN is forced the maximize the usefulness of every single parameter to get the same result. It doesn't have to worry about overfitting. I am not sure if the paper accounted for this though, it's been awhile since I read it.
  - sgt101 12y ago
    Just checking what you are saying in your last paragraph: do you mean to say that shallow NN's don't overfit?
    - Houshalter 12y ago
      If I understand correctly, they generated artificial data from the NN they were mimicking.
    - wodenokoto 12y ago
      No, shallow NN definitely overfits in normal circumstances. However, in this case they are building a shallow NN that is not learning a normal function, but is being trained to emulate a deep network of equivalent amount of parameters. Parents claim is that in that case, we don't have to worry about over fitting, since we want every result of target network to be as closely emulated as possible in the shallow network.
dougabug 12y ago
It seems reasonable that when you decompose the representation of the target function you are trying to learn into more layers, each of the layers can be "more gently" modulated to emulate the target function as observed through the training data. Differential techniques might therefore more readily find good approximations to the target function. Once the structure has been captured by finely slicing it into more layers, this structure can be compressed down into a shallower representation.

by ivan_ah 12y ago

amalcon 12y ago

The novel part is not that the shallow network can compute the same functions (that is interesting, but that particular result is decades old), but the notion that the shallow network can learn the same functions efficiently. What we already knew: For any given neural network of N layers, there exists a 3-layer neural network computing the same function. What we didn't know: There are (at least reasonably often) ways of training shallow networks that get similar efficiency to the ways of training deep networks given the same functions.

tiger10guy 12y ago
Their method for training a shallow network requires that one first train a deep network.
Houshalter 12y ago
No, they require training a deep neural network first. Then training the shallow net to mimic the deep one. The claim is that the shallow net can have the same number of parameters as the deep net. It's always been known that a large enough shallow NN can theoretically approximate any function. But that it can do so with few parameters is very surprising. My best guess is that the majority of parameters in deep NNs are unused or redundant. I.e they have a low weight, or don't influence the output very much, or another neuron computes mostly the same function. Many types of regularization like dropOut and weight decay heavily encourage this. Whereas the shallow NN is forced the maximize the usefulness of every single parameter to get the same result. It doesn't have to worry about overfitting. I am not sure if the paper accounted for this though, it's been awhile since I read it.
- sgt101 12y ago
  Just checking what you are saying in your last paragraph: do you mean to say that shallow NN's don't overfit?
  - Houshalter 12y ago
    If I understand correctly, they generated artificial data from the NN they were mimicking.
  - wodenokoto 12y ago
    No, shallow NN definitely overfits in normal circumstances. However, in this case they are building a shallow NN that is not learning a normal function, but is being trained to emulate a deep network of equivalent amount of parameters. Parents claim is that in that case, we don't have to worry about over fitting, since we want every result of target network to be as closely emulated as possible in the shallow network.

dougabug 12y ago

It seems reasonable that when you decompose the representation of the target function you are trying to learn into more layers, each of the layers can be "more gently" modulated to emulate the target function as observed through the training data. Differential techniques might therefore more readily find good approximations to the target function. Once the structure has been captured by finely slicing it into more layers, this structure can be compressed down into a shallower representation.