3 ms·

That’s not wrong but does misrepresent history. For many years a 3 layer network (1 input, 1 hidden, 1 output) was the standard and considered sufficient, than

That’s not wrong but does misrepresent history. For many years a 3 layer network (1 input, 1 hidden, 1 output) was the standard and considered sufficient, thanks to several relevant but non constructive approximation theorems (mostly one due to Kolmogorov and one due to Cybenko, if I am not mistaken). They were also considered practically required because everyone was using sigmoids which have a vanishing gradient problem. Several things were needed to break away to where we are now: unbounded functions (like ReLU) to avoid vanishing moments; a lot more layers; a lot more parameters and compute power. When Schmidhuber (and later Hinton) showed that many-layer nets work well, that was non trivial and almost revolutionary. It is now trivial and all nets are “deep”. But that wasn’t the case when the breakthroughs were made, and the terminology stuck.

aborsy 6y ago
Take a look at the papers of Hubel and Wisel and mcculloch and Pits in 40s. If I recall correctly, you can see diagrams of a collection layered neurons. The architecture of a layer of simple cells and a layer of complex cells modeling the visual cortex is a layered architecture. The multi layer perceptron was proposed right after the perceptron. That was biological inspiration, and the analogy with layered “deep” brain. Fukushima revisited this layered architecture in the context of convolutional nnets. Cybenko,Hornik etc studied the mathematical properties of multi layer feed forward nnets late 80s. Obviously, in the field of systems control equivalent “deep learning” and “reinforcement learning “ were studied since 50s. This includes what’s called “back propagation algorithm” It was all multi layer nnets until it was simulated.
- beagle3 6y ago
  Indeed, this is all true. But do remember that the kolmogorov-arnold theorem says a 3-layer (n:2n+1:m) network is a universal continuous approximator (using an unknown neuron transfer function) -- people in the '80s were looking at 3 layer networks as sufficient partly because of that. I have no time to go look at all those sources now, but having dabbled in nets since the late '80s myself, I remember vanishing gradients were sort-of a surprise, because everyone was under the impression that simple backpropagation should just work, and it didn't. A lot of that early work you refer to was also mostly about linear transfer functions, and though the exact type of non-linearity doesn't matter, some of its properties do - and as I mentioned, sigmoids - which were all the rage in the '80s - are a dead end with the wrong kind of nonlinearity. Nothing about the structure* of multilayer models is new. But successfully training them - which didn't happen until Schmidhuber and Hinton (depends on who you ask ...) - is relatively new; and that advance is responsible for the term "deep learning". We do not disagree about the details; but we do seem to disagree about the historical context and narrative.
  - ozankabak 6y ago
    You are talking about Sprecher's modification to the original Kolmogorov-Arnold theorem, right? This version, and its implications, have been a lingering wondering for me for quite a while. Are you aware of any research on 3-layer networks where the unknown transfer function is also learnable? I suspect such an approach does not result in good models (otherwise we would have known about them!), but I can not articulate why. Where exactly does the K-A reasoning fail when we try to apply it in practice?
    - beagle3 6y ago
      Yes. I haven't touched this since 1995, so I had to refresh my memory. I was indeed, talking about Sprecher's modification. Back when I studied this, the proofs I found were not constructive. I was unaware, but apparently Gribel gave a constructive proof in 2009 (link from Wikipedia article about KA rep theorem). I would have to read it and hope I am not too rusty to understand it before I could really ponder your question... But I could offer two places I would have looked: 1. The approximation is of a continuous function, and such approximations (e.g. chebychev, bernstein) usually require that you be able to sample the function at specific points - but learning usually gives you training data that does not correspond to those specific points. It's possible that construction fails here somehow. 2. The approximation is too hard in practice. This is the too often the case for Breiman's beautiful ACE (Alternating Conditional Expectation) which, if you squint hard enough, looks like a two-layer network where each neuron has its own transfer function. The algorithm is incredibly simple in theory, but very hard to use in practice.
- im3w1l 6y ago
  You are right, that deep networks were always there. But university textbooks dismissed them, and the common thought was that they were not necessary.

aborsy 6y ago

Take a look at the papers of Hubel and Wisel and mcculloch and Pits in 40s. If I recall correctly, you can see diagrams of a collection layered neurons. The architecture of a layer of simple cells and a layer of complex cells modeling the visual cortex is a layered architecture. The multi layer perceptron was proposed right after the perceptron. That was biological inspiration, and the analogy with layered “deep” brain. Fukushima revisited this layered architecture in the context of convolutional nnets. Cybenko,Hornik etc studied the mathematical properties of multi layer feed forward nnets late 80s. Obviously, in the field of systems control equivalent “deep learning” and “reinforcement learning “ were studied since 50s. This includes what’s called “back propagation algorithm” It was all multi layer nnets until it was simulated.

beagle3 6y ago
Indeed, this is all true. But do remember that the kolmogorov-arnold theorem says a 3-layer (n:2n+1:m) network is a universal continuous approximator (using an unknown neuron transfer function) -- people in the '80s were looking at 3 layer networks as sufficient partly because of that. I have no time to go look at all those sources now, but having dabbled in nets since the late '80s myself, I remember vanishing gradients were sort-of a surprise, because everyone was under the impression that simple backpropagation should just work, and it didn't. A lot of that early work you refer to was also mostly about linear transfer functions, and though the exact type of non-linearity doesn't matter, some of its properties do - and as I mentioned, sigmoids - which were all the rage in the '80s - are a dead end with the wrong kind of nonlinearity. Nothing about the structure* of multilayer models is new. But successfully training them - which didn't happen until Schmidhuber and Hinton (depends on who you ask ...) - is relatively new; and that advance is responsible for the term "deep learning". We do not disagree about the details; but we do seem to disagree about the historical context and narrative.
- ozankabak 6y ago
  You are talking about Sprecher's modification to the original Kolmogorov-Arnold theorem, right? This version, and its implications, have been a lingering wondering for me for quite a while. Are you aware of any research on 3-layer networks where the unknown transfer function is also learnable? I suspect such an approach does not result in good models (otherwise we would have known about them!), but I can not articulate why. Where exactly does the K-A reasoning fail when we try to apply it in practice?
  - beagle3 6y ago
    Yes. I haven't touched this since 1995, so I had to refresh my memory. I was indeed, talking about Sprecher's modification. Back when I studied this, the proofs I found were not constructive. I was unaware, but apparently Gribel gave a constructive proof in 2009 (link from Wikipedia article about KA rep theorem). I would have to read it and hope I am not too rusty to understand it before I could really ponder your question... But I could offer two places I would have looked: 1. The approximation is of a continuous function, and such approximations (e.g. chebychev, bernstein) usually require that you be able to sample the function at specific points - but learning usually gives you training data that does not correspond to those specific points. It's possible that construction fails here somehow. 2. The approximation is too hard in practice. This is the too often the case for Breiman's beautiful ACE (Alternating Conditional Expectation) which, if you squint hard enough, looks like a two-layer network where each neuron has its own transfer function. The algorithm is incredibly simple in theory, but very hard to use in practice.
im3w1l 6y ago
You are right, that deep networks were always there. But university textbooks dismissed them, and the common thought was that they were not necessary.