4 ms·

Artificial neurons are a model rather than a simulation. McCullough & Pitts sought to computationally model neuron's information processing ability, not simula

by deepnet 10y ago

Artificial neurons are a model rather than a simulation. McCullough & Pitts sought to computationally model neuron's information processing ability, not simulate their biology. Their 1943 paper "A logical calculus of the ideas immanent in nervous activity" invented the Artificial neuron. "Because of the "all-or-none" character of nervous activity, neural events and the relations among them can be treated by means of propositional logic." - Mcullough & Pitts 1943 Rosenblatt's Perceptron's tunable inputs & Werbos's Backpropagation allowed the Artificial Neural Nets Hebbian learning. Minsky & Papert showed a Multi-layer perceptron can learn non-linear functions.

danielmorozoff 10y ago
It is quite interesting that you bring up Hebbian learning, which at its heart is attempting to endow a model with a form of memory stored in the connection or 'edge'. Current state of the art systems (RNNs, LSTM and ResNets) are taking this one step further attempting to create dynamic memory (much like we believe neural networks work biologically for short term memory storage). I wonder if the next steps for models are to formulate mechanisms for actually modifying their own graphical structure with a sort of 'protein' based memory, much like the brain is believed to store long term memory? Does anyone have/ or has seen any papers on the topic? Could you provide refs?
- deepnet 10y ago
  Yours is a good characterisation of Hebbian learning. Backpropagation in ANN & Hebbian learning in NN analagously strengthen correlations between inputs & succesful outputs - learning useful associations. Stanley & Riskonen's NEAT evolves neural structure http://nn.cs.utexas.edu/?neat http://nn.cs.utexas.edu/?neat Schmidhuber & Togelius have nets that prune connections to evolve / learn structure.
  - danielmorozoff 10y ago
    Thank you for the references. I was not as much interested in mechanism to reduce the training speed by optimizing the structure in order to learn more efficiently. NEAT aims to 'minimize topologies and grow incrementally.' Rather my interest was in something similar to Q-learning mechanism or RL-LSTM that could learn structural sub-domains within a network, and reimplement them in other portions during training (maybe paralleling NEAT) or during instances of poor output probability.
    - deepnet 10y ago
      There is some evidence that large deep nets evolve using convergent strategies and solutions across modalities - at least for words and pictures. Low level specific edge & feature detectors combine into parts of things, these then combine into thing detectors and ultimately vectors representing objects and encoding their relationships. The final layer classification ( the training objective ) is a low dimensional slice of a rich conceptually navigable semantic vector space. That these top layer 'thought vectors' occur even for limited classification objectives implies the structure comes from the datas contextual relationships and the net learns how things relate to better classify what things are.

by deepnet 10y ago

danielmorozoff 10y ago

It is quite interesting that you bring up Hebbian learning, which at its heart is attempting to endow a model with a form of memory stored in the connection or 'edge'. Current state of the art systems (RNNs, LSTM and ResNets) are taking this one step further attempting to create dynamic memory (much like we believe neural networks work biologically for short term memory storage). I wonder if the next steps for models are to formulate mechanisms for actually modifying their own graphical structure with a sort of 'protein' based memory, much like the brain is believed to store long term memory? Does anyone have/ or has seen any papers on the topic? Could you provide refs?

deepnet 10y ago
Yours is a good characterisation of Hebbian learning. Backpropagation in ANN & Hebbian learning in NN analagously strengthen correlations between inputs & succesful outputs - learning useful associations. Stanley & Riskonen's NEAT evolves neural structure http://nn.cs.utexas.edu/?neat http://nn.cs.utexas.edu/?neat Schmidhuber & Togelius have nets that prune connections to evolve / learn structure.
- danielmorozoff 10y ago
  Thank you for the references. I was not as much interested in mechanism to reduce the training speed by optimizing the structure in order to learn more efficiently. NEAT aims to 'minimize topologies and grow incrementally.' Rather my interest was in something similar to Q-learning mechanism or RL-LSTM that could learn structural sub-domains within a network, and reimplement them in other portions during training (maybe paralleling NEAT) or during instances of poor output probability.
  - deepnet 10y ago
    There is some evidence that large deep nets evolve using convergent strategies and solutions across modalities - at least for words and pictures. Low level specific edge & feature detectors combine into parts of things, these then combine into thing detectors and ultimately vectors representing objects and encoding their relationships. The final layer classification ( the training objective ) is a low dimensional slice of a rich conceptually navigable semantic vector space. That these top layer 'thought vectors' occur even for limited classification objectives implies the structure comes from the datas contextual relationships and the net learns how things relate to better classify what things are.