5 ms·
My question, and something this doesn't get into, is this: how do you train a RNN?
by TheLoneWolfling 11y ago
My question, and something this doesn't get into, is this: how do you train a RNN?
- deepnet 11y agoYou need an error signal - a target value is compared with the networks prediction. That error is carefully assigned proportionally to the network weights that contributed to it and the weights adjusted a small amount in that direction. This is repeated many times. Backpropagation suffers from vanishing gradients on very deep neural nets. Recurrent Neural Nets can be very deep in time. Or the weights could be evolved using Genetic Programming.
- raverbashing 11y ago> Backpropagation suffers from vanishing gradients on very deep neural nets. Especially when using saturating functions (tanh/sigmoid) > Or the weights could be evolved using Genetic Programming GA, not GP http://en.wikipedia.org/wiki/Genetic_algorithm http://en.wikipedia.org/wiki/Genetic_algorithm
- skorgu 11y agoIt would be interesting to occasionally train the generated C against a compiler.
- Houshalter 11y agoIt would also be ideal to use a higher level interpreted language, and have it try to generate one page scripts rather than giant mega projects like linux.
- deepnet 11y agoIn "Learning to Execute" by Zaremba & Sutskever http://arxiv.org/abs/1410.4615 http://arxiv.org/abs/1410.4615 An RNN learns snippets of python Their next paper is "Reinforcement Learning Neural Turing Machines" http://arxiv.org/abs/1505.00521 http://arxiv.org/abs/1505.00521 based on Graves "Neural Turing Machines" http://arxiv.org/abs/1410.5401 http://arxiv.org/abs/1410.5401, which attempts to infer algorithms from the result. In a lost BBC interview from 1951 Turing reputedly spoke of evolving cpu bitmasks for computation.
- warfangle 11y ago> Or the weights could be evolved using Genetic Programming. Some algorithms, such as NEAT[0], use a genetic algorithm to describe not only the weights on edges in the network, but also the shape of the network itself - e.g., instead of every node of one layer connected to every node of the next, only certain connections are made. 0. http://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies http://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_to...
- egocodedinsol 11y agothere are various ways, but one is to unroll the network about some timestep and treat it as a regular NN. You might find this helpful: http://minds.jacobs-university.de/sites/default/files/uploads/papers/ESNTutorialRev.pdf http://minds.jacobs-university.de/sites/default/files/upload...
- TheLoneWolfling 11y agoThat sounds rather absurdly computationally expensive. Thanks for the link, I'll take a look.