4 ms·

These two comments seem to contradict each other: "we have no idea how neurons are storing information, how they are computing, what the rules are, what the al

These two comments seem to contradict each other: "we have no idea how neurons are storing information, how they are computing, what the rules are, what the algorithms are, what the representations are, and the like." "...you get an output from the end of the layers, and you propagate a signal backwards through the layers to change all the parameters. It’s pretty clear the brain doesn’t do something like that. " So why can't the brain do some kind of backpropagation?

craigyk 12y ago
I agree. Neuroscience isn't my specialty, but I don't remember hearing about research showing that the brain does not use feedback... Overall though, I think there is a lot of truth to his message.
- scottlocklin 12y ago
  Think about it; biological neurons most definitely don't use backprop. Physical neurons can't backprop linearly if they forward prop nonlinearly. I think Hinton pointed this out, though almost all ANN researchers agree. Hebbian learning .... maybe. I never thought too much about contrastive divergence. It doesn't really matter; brains most certainly don't work like ANNs, except maybe in some weird mean field sense for a few things like liquid state machines. It would be a huge coincidence if LeCun or Hinton or whoever magically wrote down the brain equation....
jerf 12y ago
We know the brain doesn't do "backpropogation" in the sense that backpropagation is a complete description of how the brain works. It may be a component, but there has to be much more, because what backpropogation can do is fairly well characterized mathematically now and is not sufficient to build anything like a human brain. If you haven't seen a paper that clearly spells that out, it's because it would be considered too trivially obvious a result to publish. When we don't know how a thing works that does not preclude us being able to eliminate some of the possibilities. Our ignorance is profound, but not total.
- deleted 12y ago
  [deleted]
robg 12y ago
The brain could, and likely does, individual neurons do not. Synaptic current only travels in one direction.
- blennon 12y ago
  This is not entirely correct. There is a lot of evidence that action potentials backpropagate into the dendrites [1]. On a local level, this allows synapses on the dendrites to "be aware" of the activity of the post-synaptic neuron. However, I do not think this accomplishes/implements the ANN backpropagation algorithm (although a few statements and a citation in [1] alludes to this being possible). [1] http://en.wikipedia.org/wiki/Neural_backpropagation http://en.wikipedia.org/wiki/Neural_backpropagation
  - robg 12y ago
    Yes, that is a very interesting complication to the standard model.
sherjilozair 12y ago
This is a reply to multiple sibling comments. There is actually recent work which shows that deep learning methods can also work WITHOUT any reverse signals: http://s.yosinski.com/dan_cownden_presentation.pdf http://s.yosinski.com/dan_cownden_presentation.pdf
- jimduk 12y ago
  Interesting paper. Any more details on the architecture of the feedback connections. Also I can't tell from the paper where and how weights are being updated e..g what does "train" mean in this context
  - asdavis 12y ago
    I believe instead of multiplying the delta by W^t to backpropagate the error from layer l to l-1, you multiply it by a random projection B. It's hard to dig deeper because there doesn't appear to be any other information on it except here: http://isis-innovation.com/licence-details/accelerating-machine-learning/ http://isis-innovation.com/licence-details/accelerating-mach... As an aside, are they really trying to patent a slight twist on backpropagation? That seems pretty counter-productive to me.
- asdavis 12y ago
  It looks like there are still reverse signals (eg., deltas), but they are multiplied by a random matrix B instead of the usual transpose of the weight matrix for that layer. Am I misunderstanding?
pessimizer 12y ago
As a weak epiphenominalist, I'd argue that all conscious thought is a backpropagation. Is it controversial that the brain is able to perceive its own output, or am I misunderstanding how backpropagation in neural networks is implemented?
- dnautics 12y ago
  that's not what backpropagation is. Backpropagation is best thought of as a 'cheat' (algorithmic simplification) that allows you to calculate the derivative of a feed-forward neural net. You would need to calculate the derivative of a neural net in general to optimize relative to some cost function, you could use, for example, gradient descent, but that is computationally costly. For some neural nets, you still have a gradient, but the concept of back or forward propagation is not defineable. Based on the topology and structure of biological neural nets, what would you think is the case?

craigyk 12y ago

I agree. Neuroscience isn't my specialty, but I don't remember hearing about research showing that the brain does not use feedback... Overall though, I think there is a lot of truth to his message.

scottlocklin 12y ago
Think about it; biological neurons most definitely don't use backprop. Physical neurons can't backprop linearly if they forward prop nonlinearly. I think Hinton pointed this out, though almost all ANN researchers agree. Hebbian learning .... maybe. I never thought too much about contrastive divergence. It doesn't really matter; brains most certainly don't work like ANNs, except maybe in some weird mean field sense for a few things like liquid state machines. It would be a huge coincidence if LeCun or Hinton or whoever magically wrote down the brain equation....

jerf 12y ago

We know the brain doesn't do "backpropogation" in the sense that backpropagation is a complete description of how the brain works. It may be a component, but there has to be much more, because what backpropogation can do is fairly well characterized mathematically now and is not sufficient to build anything like a human brain. If you haven't seen a paper that clearly spells that out, it's because it would be considered too trivially obvious a result to publish. When we don't know how a thing works that does not preclude us being able to eliminate some of the possibilities. Our ignorance is profound, but not total.

deleted 12y ago
[deleted]

robg 12y ago

The brain could, and likely does, individual neurons do not. Synaptic current only travels in one direction.

blennon 12y ago
This is not entirely correct. There is a lot of evidence that action potentials backpropagate into the dendrites [1]. On a local level, this allows synapses on the dendrites to "be aware" of the activity of the post-synaptic neuron. However, I do not think this accomplishes/implements the ANN backpropagation algorithm (although a few statements and a citation in [1] alludes to this being possible). [1] http://en.wikipedia.org/wiki/Neural_backpropagation http://en.wikipedia.org/wiki/Neural_backpropagation
- robg 12y ago
  Yes, that is a very interesting complication to the standard model.

sherjilozair 12y ago

This is a reply to multiple sibling comments. There is actually recent work which shows that deep learning methods can also work WITHOUT any reverse signals: http://s.yosinski.com/dan_cownden_presentation.pdf http://s.yosinski.com/dan_cownden_presentation.pdf

jimduk 12y ago
Interesting paper. Any more details on the architecture of the feedback connections. Also I can't tell from the paper where and how weights are being updated e..g what does "train" mean in this context
- asdavis 12y ago
  I believe instead of multiplying the delta by W^t to backpropagate the error from layer l to l-1, you multiply it by a random projection B. It's hard to dig deeper because there doesn't appear to be any other information on it except here: http://isis-innovation.com/licence-details/accelerating-machine-learning/ http://isis-innovation.com/licence-details/accelerating-mach... As an aside, are they really trying to patent a slight twist on backpropagation? That seems pretty counter-productive to me.
asdavis 12y ago
It looks like there are still reverse signals (eg., deltas), but they are multiplied by a random matrix B instead of the usual transpose of the weight matrix for that layer. Am I misunderstanding?

pessimizer 12y ago

As a weak epiphenominalist, I'd argue that all conscious thought is a backpropagation. Is it controversial that the brain is able to perceive its own output, or am I misunderstanding how backpropagation in neural networks is implemented?

dnautics 12y ago
that's not what backpropagation is. Backpropagation is best thought of as a 'cheat' (algorithmic simplification) that allows you to calculate the derivative of a feed-forward neural net. You would need to calculate the derivative of a neural net in general to optimize relative to some cost function, you could use, for example, gradient descent, but that is computationally costly. For some neural nets, you still have a gradient, but the concept of back or forward propagation is not defineable. Based on the topology and structure of biological neural nets, what would you think is the case?