4 ms·

I am simplifying it to a level where it becomes borderline useless/wrong, but it should give a gist of the idea: it turns out that frequently you do not need to

I am simplifying it to a level where it becomes borderline useless/wrong, but it should give a gist of the idea: it turns out that frequently you do not need to train anything but the last layer of a deep neural network, if all the other layers are sufficiently big arbitrary and weird. In reservoir computing you replace all but the last layer of the network with a "dynamical system", i.e. a large and meaningless map.

deleted 8y ago
[deleted]
marviel 8y ago
Thanks for the eli5! Helpful
gbrown 8y ago
So treat it like a basis expansion?
- GlenTheMachine 8y ago
  If I understand the question correctly, ie is a reservoir computing approach simply projecting an input vector into a higher dimensional space, like a wonky support vector machine, then I think the answer is: unclear. Reservoir computing approaches usually have the intermediate layers be recurrent, eg they implement a dynamical system. Theoretically, this is actually Turing complete, although good luck programming it. But in any case, the range of behaviors — transformations in the data — that a dynamical system can implement is much more powerful than just a basis expansion. However, whether this is what is happening in actual practice is really, really unclear to me. A lot of recurrent neural networks aren't doing anything more than an equivalent feedforward network, and the same thing may be true here: reservoir approaches might, for the most part, really just be performing nonlinear projection to higher dimensional spaces, and then the output layer is being trained to classify those patterns.
  - gbrown 8y ago
    Thanks, that makes sense.
AllegedAlec 8y ago
I may be having some misunderstanding here, but with a large enough set of random intermediate layers, wouldn't yet expect that some of the connections would accidentally be really good signal carriers, and that by training the last layer, you're basically training it to find the good signal carriers in between all the noise?

deleted 8y ago

[deleted]

marviel 8y ago

Thanks for the eli5! Helpful

gbrown 8y ago

So treat it like a basis expansion?

GlenTheMachine 8y ago
If I understand the question correctly, ie is a reservoir computing approach simply projecting an input vector into a higher dimensional space, like a wonky support vector machine, then I think the answer is: unclear. Reservoir computing approaches usually have the intermediate layers be recurrent, eg they implement a dynamical system. Theoretically, this is actually Turing complete, although good luck programming it. But in any case, the range of behaviors — transformations in the data — that a dynamical system can implement is much more powerful than just a basis expansion. However, whether this is what is happening in actual practice is really, really unclear to me. A lot of recurrent neural networks aren't doing anything more than an equivalent feedforward network, and the same thing may be true here: reservoir approaches might, for the most part, really just be performing nonlinear projection to higher dimensional spaces, and then the output layer is being trained to classify those patterns.
- gbrown 8y ago
  Thanks, that makes sense.

AllegedAlec 8y ago

I may be having some misunderstanding here, but with a large enough set of random intermediate layers, wouldn't yet expect that some of the connections would accidentally be really good signal carriers, and that by training the last layer, you're basically training it to find the good signal carriers in between all the noise?