6 ms·

It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we ar

It never ceases to amaze me that the best steps towards achieving AI is to look at how we perceive that a Neuron works and simulate it. And the thing is, we aren't exactly sure why exactly that is.. it's amazing. Sometimes the best thing we can do is imitate nature

spott 10y ago
This isn't strictly true though. Spiking Neural Networks [0] attempt to be more accurate representations of human neurons, but haven't really caught on because they aren't really much better than our perceptron model of neurons, at least for the things we are trying to do with them. [0]http://www.ane.pl/pdf/7146.pdf http://www.ane.pl/pdf/7146.pdf
- joantune 10y ago
  nice, thanks for sharing, interesting read so far (read the introduction), will definitely give it a better look out of curiosity
spynxic 10y ago
I find that somewhat strange. Why attribute the idea of introducing new nodes to a graph to biological concepts? It seems like a simple step in exploration, similar to how one might think to vary the weights of the nodes randomly over some range.. unless there is some technique biology uses to pre-configure the nodes upon introduction to the network, that might be rather interesting.
- joantune 10y ago
  Because they tried to model neurogenesis, the same way that artificial neural networks were invented while trying to mimic some parts of how neurons work I guess..
deleted 10y ago
[deleted]
partycoder 10y ago
Well, neurons have many properties... their information processing capabilities are one aspect, but they also deal with the physical level of communication and staying healthy. Neurons are also a family of cells, and are very diverse in shapes and functions. We tend to oversimplify our representation of neurons. There are simple neurons and then you have neurons like the Purkinje cell that are massive. Neurons also rely on their counterparts, the glial cells, that are much less often mentioned. I think because of this, it will be a while until we fully understand the role of each one of them.
- joantune 10y ago
  If i'm not mistaken (I had the introductory class on Neural networks quite some years ago) this all started out of trying to simulate neurons in a manner. Much so that one of the pioneers of this got discredited by other scientist that for some reason simply could not accept that these would work, and that same pioneer started to get his funding discredited and started believing in his opposition so much so that he sailed away to his death, some argue intentionally (as his life's work had been, even by his eyes, seen as useless). Now, I'm having a hard time remembering the names of the people on that story, if someone knows who I'm talking about, please remind me of those
  - partycoder 10y ago
    Probably you might be referring to McCulloch and Pitts.
throwaway287391 10y ago
I completely blame my own community, rather than you, for writing this, but as an AI researcher, your comment is terribly painful to read. We have little to no idea how actual neurons (let alone entire brains) really work. The things that are often called "(artificial) neural networks" really shouldn't be called that. I strongly prefer terms like "computational networks" or (where applicable) "recurrent/convolutional networks".
- joantune 10y ago
  I meant that it's a bit similar in the way that the passing on of signals and how over time neurons prefer some connections rather than others. This part is a bit similar, no?
  - bmh100 10y ago
    In a very hand-wavy sense, yes. The same can be said of paths to food by ant colonies. The way that ANNs have been drawn as circles with arrows between them looks like a cartoon version of neurons and synapses, which is the origin of the "neural network" part. The timing of data from hidden node to hidden node, the activation functions, and the hidden node outputs have very little to do with biological neurons. ANNs have more in common with a CPU than a brain.
    - joantune 10y ago
      Wasn't the attempt of modeling a collective of neurons, their synapsis, and the way that some connections are reinforced the genesis of the artificial neural networks? That's how the first person brought that concept to life, no? He didn't even have a theoretical explanation on how/why that would work for something, right? > ANNs have more in common with a CPU than a brain How so? which parts are similar?
      - bmh100 10y ago
        Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs. slow activation speed (Hz) * The use of negative signals vs. strictly positive quantities of neurotransmitters * Low average connections (10-1000) vs. high average connections (5,000-100,000) * Low energy efficiency vs. high energy efficiency For a detailed essay on the topic, see: http://timdettmers.com/2015/07/27/brain-vs-deep-learning-singularity/ http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin...
visarga 10y ago
We're not simulating brain neurons: - real neurons are stochastic and communicate through spikes, artificial neurons can communicate real values efficiently - real neurons are more like automatons, they have a dynamic in time, learning happens as a continuous interaction with only its neighbors; artificial neurons are "static" (use discrete time) and implemented by forward and backward pass, and also can use nonlocal information - real neurons can't backpropagate, because backprop requires the transmission of gradients back the same connections, but in reverse - brain connections don't support that kind of bidirectional data flow; artificial neurons work best by backprop - real neurons can't implement convolutions, it would require a neuron to slide over a field; also real neurons can't implement RNNs as they are, and don't use backpropagation through time BPTT So, artificial neurons are much less hampered and can do many things that real neurons can't do or have to use some less efficient method. That means brain neurons still have some tricks up their sleeve. Artificial neurons are quite different from brain neurons, and it's right to be so, because they can be more efficient that way.

spott 10y ago

This isn't strictly true though. Spiking Neural Networks [0] attempt to be more accurate representations of human neurons, but haven't really caught on because they aren't really much better than our perceptron model of neurons, at least for the things we are trying to do with them. [0]http://www.ane.pl/pdf/7146.pdf http://www.ane.pl/pdf/7146.pdf

joantune 10y ago
nice, thanks for sharing, interesting read so far (read the introduction), will definitely give it a better look out of curiosity

spynxic 10y ago

I find that somewhat strange. Why attribute the idea of introducing new nodes to a graph to biological concepts? It seems like a simple step in exploration, similar to how one might think to vary the weights of the nodes randomly over some range.. unless there is some technique biology uses to pre-configure the nodes upon introduction to the network, that might be rather interesting.

joantune 10y ago
Because they tried to model neurogenesis, the same way that artificial neural networks were invented while trying to mimic some parts of how neurons work I guess..

deleted 10y ago

[deleted]

partycoder 10y ago

Well, neurons have many properties... their information processing capabilities are one aspect, but they also deal with the physical level of communication and staying healthy. Neurons are also a family of cells, and are very diverse in shapes and functions. We tend to oversimplify our representation of neurons. There are simple neurons and then you have neurons like the Purkinje cell that are massive. Neurons also rely on their counterparts, the glial cells, that are much less often mentioned. I think because of this, it will be a while until we fully understand the role of each one of them.

joantune 10y ago
If i'm not mistaken (I had the introductory class on Neural networks quite some years ago) this all started out of trying to simulate neurons in a manner. Much so that one of the pioneers of this got discredited by other scientist that for some reason simply could not accept that these would work, and that same pioneer started to get his funding discredited and started believing in his opposition so much so that he sailed away to his death, some argue intentionally (as his life's work had been, even by his eyes, seen as useless). Now, I'm having a hard time remembering the names of the people on that story, if someone knows who I'm talking about, please remind me of those
- partycoder 10y ago
  Probably you might be referring to McCulloch and Pitts.

throwaway287391 10y ago

I completely blame my own community, rather than you, for writing this, but as an AI researcher, your comment is terribly painful to read. We have little to no idea how actual neurons (let alone entire brains) really work. The things that are often called "(artificial) neural networks" really shouldn't be called that. I strongly prefer terms like "computational networks" or (where applicable) "recurrent/convolutional networks".

joantune 10y ago
I meant that it's a bit similar in the way that the passing on of signals and how over time neurons prefer some connections rather than others. This part is a bit similar, no?
- bmh100 10y ago
  In a very hand-wavy sense, yes. The same can be said of paths to food by ant colonies. The way that ANNs have been drawn as circles with arrows between them looks like a cartoon version of neurons and synapses, which is the origin of the "neural network" part. The timing of data from hidden node to hidden node, the activation functions, and the hidden node outputs have very little to do with biological neurons. ANNs have more in common with a CPU than a brain.
  - joantune 10y ago
    Wasn't the attempt of modeling a collective of neurons, their synapsis, and the way that some connections are reinforced the genesis of the artificial neural networks? That's how the first person brought that concept to life, no? He didn't even have a theoretical explanation on how/why that would work for something, right? > ANNs have more in common with a CPU than a brain How so? which parts are similar?
    - bmh100 10y ago
      Yes, ANNs are inspired by the brain. Here is a list of properties that ANNs shared with CPUs that are different from brains: * Synchronized activation vs. asynchronous / partially synchronous activation * Digital signals vs. analog signals * Instantaneous transmission of signals vs. delay imposed by axon and dendrite length * Uniform signal vs. use of various neurotransmitter signals * Rapid activation speed (GHz) vs. slow activation speed (Hz) * The use of negative signals vs. strictly positive quantities of neurotransmitters * Low average connections (10-1000) vs. high average connections (5,000-100,000) * Low energy efficiency vs. high energy efficiency For a detailed essay on the topic, see: http://timdettmers.com/2015/07/27/brain-vs-deep-learning-singularity/ http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin...

visarga 10y ago

We're not simulating brain neurons: - real neurons are stochastic and communicate through spikes, artificial neurons can communicate real values efficiently - real neurons are more like automatons, they have a dynamic in time, learning happens as a continuous interaction with only its neighbors; artificial neurons are "static" (use discrete time) and implemented by forward and backward pass, and also can use nonlocal information - real neurons can't backpropagate, because backprop requires the transmission of gradients back the same connections, but in reverse - brain connections don't support that kind of bidirectional data flow; artificial neurons work best by backprop - real neurons can't implement convolutions, it would require a neuron to slide over a field; also real neurons can't implement RNNs as they are, and don't use backpropagation through time BPTT So, artificial neurons are much less hampered and can do many things that real neurons can't do or have to use some less efficient method. That means brain neurons still have some tricks up their sleeve. Artificial neurons are quite different from brain neurons, and it's right to be so, because they can be more efficient that way.