5 ms·

It is quite interesting that you bring up Hebbian learning, which at its heart is attempting to endow a model with a form of memory stored in the connection or

It is quite interesting that you bring up Hebbian learning, which at its heart is attempting to endow a model with a form of memory stored in the connection or 'edge'. Current state of the art systems (RNNs, LSTM and ResNets) are taking this one step further attempting to create dynamic memory (much like we believe neural networks work biologically for short term memory storage). I wonder if the next steps for models are to formulate mechanisms for actually modifying their own graphical structure with a sort of 'protein' based memory, much like the brain is believed to store long term memory? Does anyone have/ or has seen any papers on the topic? Could you provide refs?

deepnet 10y ago
Yours is a good characterisation of Hebbian learning. Backpropagation in ANN & Hebbian learning in NN analagously strengthen correlations between inputs & succesful outputs - learning useful associations. Stanley & Riskonen's NEAT evolves neural structure http://nn.cs.utexas.edu/?neat http://nn.cs.utexas.edu/?neat Schmidhuber & Togelius have nets that prune connections to evolve / learn structure.
- danielmorozoff 10y ago
  Thank you for the references. I was not as much interested in mechanism to reduce the training speed by optimizing the structure in order to learn more efficiently. NEAT aims to 'minimize topologies and grow incrementally.' Rather my interest was in something similar to Q-learning mechanism or RL-LSTM that could learn structural sub-domains within a network, and reimplement them in other portions during training (maybe paralleling NEAT) or during instances of poor output probability.
  - deepnet 10y ago
    There is some evidence that large deep nets evolve using convergent strategies and solutions across modalities - at least for words and pictures. Low level specific edge & feature detectors combine into parts of things, these then combine into thing detectors and ultimately vectors representing objects and encoding their relationships. The final layer classification ( the training objective ) is a low dimensional slice of a rich conceptually navigable semantic vector space. That these top layer 'thought vectors' occur even for limited classification objectives implies the structure comes from the datas contextual relationships and the net learns how things relate to better classify what things are.

deepnet 10y ago

Yours is a good characterisation of Hebbian learning. Backpropagation in ANN & Hebbian learning in NN analagously strengthen correlations between inputs & succesful outputs - learning useful associations. Stanley & Riskonen's NEAT evolves neural structure http://nn.cs.utexas.edu/?neat http://nn.cs.utexas.edu/?neat Schmidhuber & Togelius have nets that prune connections to evolve / learn structure.

danielmorozoff 10y ago
Thank you for the references. I was not as much interested in mechanism to reduce the training speed by optimizing the structure in order to learn more efficiently. NEAT aims to 'minimize topologies and grow incrementally.' Rather my interest was in something similar to Q-learning mechanism or RL-LSTM that could learn structural sub-domains within a network, and reimplement them in other portions during training (maybe paralleling NEAT) or during instances of poor output probability.
- deepnet 10y ago
  There is some evidence that large deep nets evolve using convergent strategies and solutions across modalities - at least for words and pictures. Low level specific edge & feature detectors combine into parts of things, these then combine into thing detectors and ultimately vectors representing objects and encoding their relationships. The final layer classification ( the training objective ) is a low dimensional slice of a rich conceptually navigable semantic vector space. That these top layer 'thought vectors' occur even for limited classification objectives implies the structure comes from the datas contextual relationships and the net learns how things relate to better classify what things are.