5 ms·

As soon as I saw the word Deep I stopped continuing. Nnets have always been multi layer since they were invented. That’s the whole idea of progressive feature

by aborsy 6y ago

As soon as I saw the word Deep I stopped continuing. Nnets have always been multi layer since they were invented. That’s the whole idea of progressive feature extraction, and the analogy with biological brain. Theoreticians referred to them properly as nnets or multilayer nnets. Later experimentalists simulated them, thanks to the availability of the computing resources, and experimentally verified that a multi layer nnet can be more efficient than a single layer one. They added superficial terms “deep” and “AI,” “singularity,” etc., which the media and tech industry amplified for obvious reasons.

notsuoh 6y ago
Forgive me, but that seems like an excessively pedantic outright dismissal of what I am seeing to be the presentation of interesting results in the field. Whatever you want to call it—deep learning, multilayer nnets, whatever—it’s the same thing and results are exciting. Also, I believe that there is a difference between deep learning and multilayer nnets (though I can't quite articulate it), but throwing in things like “singularity” is a bit much as it’s not really related at all to the math results being presented.
- deleted 6y ago
  [deleted]
Der_Einzige 6y ago
While in practice this is true, a lot of work is being done to try to figure out how to make shallow neural networks train as fast as deep ones. We know by the UAT that in principle a shallow network with similar paramaters counts to a deep one should be capable of learning the same decision boundary... https://en.m.wikipedia.org/wiki/Universal_approximation_theorem https://en.m.wikipedia.org/wiki/Universal_approximation_theo...
- aborsy 6y ago
  It depends on the function that you are trying to approximate. I can give you a function that a shallow nnet would approximate better and functions that deep nets approximate exponentially better even with one more layer (in terms of number of neurons n). In the limit n->\infty, both reach arbitrary small errors (obviously often with different number of parameters).
  - falcor84 6y ago
    Is there a particular theorem that you're invoking here?
beagle3 6y ago
That’s not wrong but does misrepresent history. For many years a 3 layer network (1 input, 1 hidden, 1 output) was the standard and considered sufficient, thanks to several relevant but non constructive approximation theorems (mostly one due to Kolmogorov and one due to Cybenko, if I am not mistaken). They were also considered practically required because everyone was using sigmoids which have a vanishing gradient problem. Several things were needed to break away to where we are now: unbounded functions (like ReLU) to avoid vanishing moments; a lot more layers; a lot more parameters and compute power. When Schmidhuber (and later Hinton) showed that many-layer nets work well, that was non trivial and almost revolutionary. It is now trivial and all nets are “deep”. But that wasn’t the case when the breakthroughs were made, and the terminology stuck.
- aborsy 6y ago
  Take a look at the papers of Hubel and Wisel and mcculloch and Pits in 40s. If I recall correctly, you can see diagrams of a collection layered neurons. The architecture of a layer of simple cells and a layer of complex cells modeling the visual cortex is a layered architecture. The multi layer perceptron was proposed right after the perceptron. That was biological inspiration, and the analogy with layered “deep” brain. Fukushima revisited this layered architecture in the context of convolutional nnets. Cybenko,Hornik etc studied the mathematical properties of multi layer feed forward nnets late 80s. Obviously, in the field of systems control equivalent “deep learning” and “reinforcement learning “ were studied since 50s. This includes what’s called “back propagation algorithm” It was all multi layer nnets until it was simulated.
  - beagle3 6y ago
    Indeed, this is all true. But do remember that the kolmogorov-arnold theorem says a 3-layer (n:2n+1:m) network is a universal continuous approximator (using an unknown neuron transfer function) -- people in the '80s were looking at 3 layer networks as sufficient partly because of that. I have no time to go look at all those sources now, but having dabbled in nets since the late '80s myself, I remember vanishing gradients were sort-of a surprise, because everyone was under the impression that simple backpropagation should just work, and it didn't. A lot of that early work you refer to was also mostly about linear transfer functions, and though the exact type of non-linearity doesn't matter, some of its properties do - and as I mentioned, sigmoids - which were all the rage in the '80s - are a dead end with the wrong kind of nonlinearity. Nothing about the structure* of multilayer models is new. But successfully training them - which didn't happen until Schmidhuber and Hinton (depends on who you ask ...) - is relatively new; and that advance is responsible for the term "deep learning". We do not disagree about the details; but we do seem to disagree about the historical context and narrative.
marcinzm 6y ago
Multi-layer does not convey how many layers (3? 12? 100?) while deep does somewhat. If someone is talking about networks with a certain large-ish number of layers then deep seems a better shorthand to use.

by aborsy 6y ago

notsuoh 6y ago

Forgive me, but that seems like an excessively pedantic outright dismissal of what I am seeing to be the presentation of interesting results in the field. Whatever you want to call it—deep learning, multilayer nnets, whatever—it’s the same thing and results are exciting. Also, I believe that there is a difference between deep learning and multilayer nnets (though I can't quite articulate it), but throwing in things like “singularity” is a bit much as it’s not really related at all to the math results being presented.

deleted 6y ago
[deleted]

Der_Einzige 6y ago

While in practice this is true, a lot of work is being done to try to figure out how to make shallow neural networks train as fast as deep ones. We know by the UAT that in principle a shallow network with similar paramaters counts to a deep one should be capable of learning the same decision boundary... https://en.m.wikipedia.org/wiki/Universal_approximation_theorem https://en.m.wikipedia.org/wiki/Universal_approximation_theo...

aborsy 6y ago
It depends on the function that you are trying to approximate. I can give you a function that a shallow nnet would approximate better and functions that deep nets approximate exponentially better even with one more layer (in terms of number of neurons n). In the limit n->\infty, both reach arbitrary small errors (obviously often with different number of parameters).
- falcor84 6y ago
  Is there a particular theorem that you're invoking here?

beagle3 6y ago

That’s not wrong but does misrepresent history. For many years a 3 layer network (1 input, 1 hidden, 1 output) was the standard and considered sufficient, thanks to several relevant but non constructive approximation theorems (mostly one due to Kolmogorov and one due to Cybenko, if I am not mistaken). They were also considered practically required because everyone was using sigmoids which have a vanishing gradient problem. Several things were needed to break away to where we are now: unbounded functions (like ReLU) to avoid vanishing moments; a lot more layers; a lot more parameters and compute power. When Schmidhuber (and later Hinton) showed that many-layer nets work well, that was non trivial and almost revolutionary. It is now trivial and all nets are “deep”. But that wasn’t the case when the breakthroughs were made, and the terminology stuck.

aborsy 6y ago
Take a look at the papers of Hubel and Wisel and mcculloch and Pits in 40s. If I recall correctly, you can see diagrams of a collection layered neurons. The architecture of a layer of simple cells and a layer of complex cells modeling the visual cortex is a layered architecture. The multi layer perceptron was proposed right after the perceptron. That was biological inspiration, and the analogy with layered “deep” brain. Fukushima revisited this layered architecture in the context of convolutional nnets. Cybenko,Hornik etc studied the mathematical properties of multi layer feed forward nnets late 80s. Obviously, in the field of systems control equivalent “deep learning” and “reinforcement learning “ were studied since 50s. This includes what’s called “back propagation algorithm” It was all multi layer nnets until it was simulated.
- beagle3 6y ago
  Indeed, this is all true. But do remember that the kolmogorov-arnold theorem says a 3-layer (n:2n+1:m) network is a universal continuous approximator (using an unknown neuron transfer function) -- people in the '80s were looking at 3 layer networks as sufficient partly because of that. I have no time to go look at all those sources now, but having dabbled in nets since the late '80s myself, I remember vanishing gradients were sort-of a surprise, because everyone was under the impression that simple backpropagation should just work, and it didn't. A lot of that early work you refer to was also mostly about linear transfer functions, and though the exact type of non-linearity doesn't matter, some of its properties do - and as I mentioned, sigmoids - which were all the rage in the '80s - are a dead end with the wrong kind of nonlinearity. Nothing about the structure* of multilayer models is new. But successfully training them - which didn't happen until Schmidhuber and Hinton (depends on who you ask ...) - is relatively new; and that advance is responsible for the term "deep learning". We do not disagree about the details; but we do seem to disagree about the historical context and narrative.

marcinzm 6y ago

Multi-layer does not convey how many layers (3? 12? 100?) while deep does somewhat. If someone is talking about networks with a certain large-ish number of layers then deep seems a better shorthand to use.