3 ms·

Perceptrons – the most basic form of a neural network

116 points by kawera 10y ago

pessimizer 10y ago
This is a question for anyone who knows more (or reads more) about neural networks than me. My problem with the perceptron since I was a kid is that it doesn't seem to model how neurons work very well at all. It creates a neuron that exists in a timeless place; kind of a homo economicus for cognition. My understanding of neurons: The signals arrive at the dendrites at different times, as soon as a signal arrives its strength decays steadily, and when the sum of decaying signals and an additional signal reach a threshold, the axon fires, getting rid of all energy and beginning the process again (creating the largely irregular pops that everybody is used to listening to.) Aside from that, there are effects from other bodies and chemical concentrations between neurons that are mostly unknown (that I get leaving out as not central to the process.) My question: is there any paper that explains how that process, which seems pretty easy to model, can be mathematically reduced to perceptrons (hopefully comprehensible to the educated layman) which are just addition and a condition?
- hodwik2 10y ago
  There are arguments that you'd need to create a computer capable of doing "infinite" precision real computation to recreate this effect.
  - pessimizer 10y ago
    Where do those arguments come from? I always thought that the problem was that it might be an irreducibly parallel process, and would need its own hardware to simulate cheaply. That does sound interesting; any links or cites?
- danielmorozoff 10y ago
  First off preceptrons are very simplified models, a first pass really at modeling brains in silico decades ago. There is a whole field of theoretical neuroscience dedicated to this problem, and have made quite a bit of progress, still a bunch of work left to be done. The classical introductory text on the topic is the book by Peter Dayan and Larry Abbott called computational and mathematical modeling of neural systems.
  - pessimizer 10y ago
    [deleted] Thanks for the reference. edit: Thanks very much; if that doesn't contain what I'm looking for, it'll give me further places to look.
- deepnet 10y ago
  Artificial neurons are a model rather than a simulation. McCullough & Pitts sought to computationally model neuron's information processing ability, not simulate their biology. Their 1943 paper "A logical calculus of the ideas immanent in nervous activity" invented the Artificial neuron. "Because of the "all-or-none" character of nervous activity, neural events and the relations among them can be treated by means of propositional logic." - Mcullough & Pitts 1943 Rosenblatt's Perceptron's tunable inputs & Werbos's Backpropagation allowed the Artificial Neural Nets Hebbian learning. Minsky & Papert showed a Multi-layer perceptron can learn non-linear functions.
  - danielmorozoff 10y ago
    It is quite interesting that you bring up Hebbian learning, which at its heart is attempting to endow a model with a form of memory stored in the connection or 'edge'. Current state of the art systems (RNNs, LSTM and ResNets) are taking this one step further attempting to create dynamic memory (much like we believe neural networks work biologically for short term memory storage). I wonder if the next steps for models are to formulate mechanisms for actually modifying their own graphical structure with a sort of 'protein' based memory, much like the brain is believed to store long term memory? Does anyone have/ or has seen any papers on the topic? Could you provide refs?
    - deepnet 10y ago
      Yours is a good characterisation of Hebbian learning. Backpropagation in ANN & Hebbian learning in NN analagously strengthen correlations between inputs & succesful outputs - learning useful associations. Stanley & Riskonen's NEAT evolves neural structure http://nn.cs.utexas.edu/?neat http://nn.cs.utexas.edu/?neat Schmidhuber & Togelius have nets that prune connections to evolve / learn structure.
      - danielmorozoff 10y ago
        Thank you for the references. I was not as much interested in mechanism to reduce the training speed by optimizing the structure in order to learn more efficiently. NEAT aims to 'minimize topologies and grow incrementally.' Rather my interest was in something similar to Q-learning mechanism or RL-LSTM that could learn structural sub-domains within a network, and reimplement them in other portions during training (maybe paralleling NEAT) or during instances of poor output probability.
- captkrag 10y ago
  There are much better models out there for modeling networks closer to the biology. The current generation is kind of understood to be an extremely rough model that draws rough inspiration from biology. But the important thing is that Ann models are performing better on image recognition and nlp tasks than any other techniques. The fact that they demonstrably work is far more important than biological mimicry. There is another model called a spiking neutral network that basically aims to mimic biology and incorporates time and usually many more neurons and sparse connectivity. While this is interesting, so far they haven't produced anything too mind blowing and due to the increased complexity they are much less computationally efficient right now. But those actually interested in the biologically inspired aspect are streare stream to
  - pessimizer 10y ago
    Thank you very much, this gives me a lot more to read.
- jm_l 10y ago
  Here's an online book I read that explains perceptrons and sigmoid neurons from the ground up. I think it's a great resource and might help answer some of your questions. http://neuralnetworksanddeeplearning.com/chap1.html http://neuralnetworksanddeeplearning.com/chap1.html This chapter also takes you through creating a handwriting recognizer in Python without having a package deal with the internals for you.
- jimfleming 10y ago
  Look into computational neuroscience, neural engineering, spiking neurons and spike-timing dependent plasticity (STDP). These are more accurate models of biological neurons and their synaptic interactions. From the Hodgkin-Huxley (HH) neuron model to integrate and fire (I&F) you can approximate and simulate various levels of realism in artificial neural networks. There's even the Nengo library[0] which can simulate these for you in Python. Unfortunately, they're very hard to work with compared to perceptrons, artificial neural networks, and deep learning (see Neural Engineering[1] for a good framework). Additionally, they're far more computationally intensive with the HH model coming in at over 800 FLOPS to the I&F model (which lacks a lot of the behavior and dynamics of biological neurons) which has 7 FLOPS. Then try representing numbers with spiking neurons... You need several dozen, working as a population[2], to accurately represent a single floating point because they're so noisy. This makes working with data pretty difficult and expensive. [0] https://github.com/nengo/nengo https://github.com/nengo/nengo [1] http://compneuro.uwaterloo.ca/research/nef.html http://compneuro.uwaterloo.ca/research/nef.html [2] https://en.wikipedia.org/wiki/Neural_coding https://en.wikipedia.org/wiki/Neural_coding
- InclinedPlane 10y ago
  Correct. The similarity between actual neurons and "neural networks" exists in the name only, functionally they are night and day different. Neural nets are really closer to multi-variate regression curves than brain matter.
gavanwoolery 10y ago
One thing the article failed to mention - which I found most interesting, is that a perceptron can function like a NAND gate, which is the building block of all other gates (AND, OR, XOR, etc) and thus can replicate the functionality of any logical array.
- supercarrot 10y ago
  Another interesting thing is that perceptron can approximate any non-linear function with just a single hidden layer. http://cstheory.stackexchange.com/questions/17545/universal-approximation-theorem-neural-networks http://cstheory.stackexchange.com/questions/17545/universal-...
appleflaxen 10y ago
I'm getting "refused to connect" DownForEveryone says it's down too. anyone have a mirror?
peternilson 10y ago
I implemented the code produced in the tutorial in python for anyone who's interested https://github.com/petenilson/pyceptron https://github.com/petenilson/pyceptron

116 points by kawera 10y ago

This is a question for anyone who knows more (or reads more) about neural networks than me. My problem with the perceptron since I was a kid is that it doesn't seem to model how neurons work very well at all. It creates a neuron that exists in a timeless place; kind of a homo economicus for cognition. My understanding of neurons: The signals arrive at the dendrites at different times, as soon as a signal arrives its strength decays steadily, and when the sum of decaying signals and an additional signal reach a threshold, the axon fires, getting rid of all energy and beginning the process again (creating the largely irregular pops that everybody is used to listening to.) Aside from that, there are effects from other bodies and chemical concentrations between neurons that are mostly unknown (that I get leaving out as not central to the process.) My question: is there any paper that explains how that process, which seems pretty easy to model, can be mathematically reduced to perceptrons (hopefully comprehensible to the educated layman) which are just addition and a condition?

hodwik2 10y ago
There are arguments that you'd need to create a computer capable of doing "infinite" precision real computation to recreate this effect.
- pessimizer 10y ago
  Where do those arguments come from? I always thought that the problem was that it might be an irreducibly parallel process, and would need its own hardware to simulate cheaply. That does sound interesting; any links or cites?
danielmorozoff 10y ago
First off preceptrons are very simplified models, a first pass really at modeling brains in silico decades ago. There is a whole field of theoretical neuroscience dedicated to this problem, and have made quite a bit of progress, still a bunch of work left to be done. The classical introductory text on the topic is the book by Peter Dayan and Larry Abbott called computational and mathematical modeling of neural systems.
- pessimizer 10y ago
  [deleted] Thanks for the reference. edit: Thanks very much; if that doesn't contain what I'm looking for, it'll give me further places to look.
deepnet 10y ago
Artificial neurons are a model rather than a simulation. McCullough & Pitts sought to computationally model neuron's information processing ability, not simulate their biology. Their 1943 paper "A logical calculus of the ideas immanent in nervous activity" invented the Artificial neuron. "Because of the "all-or-none" character of nervous activity, neural events and the relations among them can be treated by means of propositional logic." - Mcullough & Pitts 1943 Rosenblatt's Perceptron's tunable inputs & Werbos's Backpropagation allowed the Artificial Neural Nets Hebbian learning. Minsky & Papert showed a Multi-layer perceptron can learn non-linear functions.
- danielmorozoff 10y ago
  It is quite interesting that you bring up Hebbian learning, which at its heart is attempting to endow a model with a form of memory stored in the connection or 'edge'. Current state of the art systems (RNNs, LSTM and ResNets) are taking this one step further attempting to create dynamic memory (much like we believe neural networks work biologically for short term memory storage). I wonder if the next steps for models are to formulate mechanisms for actually modifying their own graphical structure with a sort of 'protein' based memory, much like the brain is believed to store long term memory? Does anyone have/ or has seen any papers on the topic? Could you provide refs?
  - deepnet 10y ago
    Yours is a good characterisation of Hebbian learning. Backpropagation in ANN & Hebbian learning in NN analagously strengthen correlations between inputs & succesful outputs - learning useful associations. Stanley & Riskonen's NEAT evolves neural structure http://nn.cs.utexas.edu/?neat http://nn.cs.utexas.edu/?neat Schmidhuber & Togelius have nets that prune connections to evolve / learn structure.
    - danielmorozoff 10y ago
      Thank you for the references. I was not as much interested in mechanism to reduce the training speed by optimizing the structure in order to learn more efficiently. NEAT aims to 'minimize topologies and grow incrementally.' Rather my interest was in something similar to Q-learning mechanism or RL-LSTM that could learn structural sub-domains within a network, and reimplement them in other portions during training (maybe paralleling NEAT) or during instances of poor output probability.
captkrag 10y ago
There are much better models out there for modeling networks closer to the biology. The current generation is kind of understood to be an extremely rough model that draws rough inspiration from biology. But the important thing is that Ann models are performing better on image recognition and nlp tasks than any other techniques. The fact that they demonstrably work is far more important than biological mimicry. There is another model called a spiking neutral network that basically aims to mimic biology and incorporates time and usually many more neurons and sparse connectivity. While this is interesting, so far they haven't produced anything too mind blowing and due to the increased complexity they are much less computationally efficient right now. But those actually interested in the biologically inspired aspect are streare stream to
- pessimizer 10y ago
  Thank you very much, this gives me a lot more to read.
jm_l 10y ago
Here's an online book I read that explains perceptrons and sigmoid neurons from the ground up. I think it's a great resource and might help answer some of your questions. http://neuralnetworksanddeeplearning.com/chap1.html http://neuralnetworksanddeeplearning.com/chap1.html This chapter also takes you through creating a handwriting recognizer in Python without having a package deal with the internals for you.
jimfleming 10y ago
Look into computational neuroscience, neural engineering, spiking neurons and spike-timing dependent plasticity (STDP). These are more accurate models of biological neurons and their synaptic interactions. From the Hodgkin-Huxley (HH) neuron model to integrate and fire (I&F) you can approximate and simulate various levels of realism in artificial neural networks. There's even the Nengo library[0] which can simulate these for you in Python. Unfortunately, they're very hard to work with compared to perceptrons, artificial neural networks, and deep learning (see Neural Engineering[1] for a good framework). Additionally, they're far more computationally intensive with the HH model coming in at over 800 FLOPS to the I&F model (which lacks a lot of the behavior and dynamics of biological neurons) which has 7 FLOPS. Then try representing numbers with spiking neurons... You need several dozen, working as a population[2], to accurately represent a single floating point because they're so noisy. This makes working with data pretty difficult and expensive. [0] https://github.com/nengo/nengo https://github.com/nengo/nengo [1] http://compneuro.uwaterloo.ca/research/nef.html http://compneuro.uwaterloo.ca/research/nef.html [2] https://en.wikipedia.org/wiki/Neural_coding https://en.wikipedia.org/wiki/Neural_coding
InclinedPlane 10y ago
Correct. The similarity between actual neurons and "neural networks" exists in the name only, functionally they are night and day different. Neural nets are really closer to multi-variate regression curves than brain matter.

gavanwoolery 10y ago

One thing the article failed to mention - which I found most interesting, is that a perceptron can function like a NAND gate, which is the building block of all other gates (AND, OR, XOR, etc) and thus can replicate the functionality of any logical array.

supercarrot 10y ago
Another interesting thing is that perceptron can approximate any non-linear function with just a single hidden layer. http://cstheory.stackexchange.com/questions/17545/universal-approximation-theorem-neural-networks http://cstheory.stackexchange.com/questions/17545/universal-...

appleflaxen 10y ago

I'm getting "refused to connect" DownForEveryone says it's down too. anyone have a mirror?

peternilson 10y ago

I implemented the code produced in the tutorial in python for anyone who's interested https://github.com/petenilson/pyceptron https://github.com/petenilson/pyceptron