6 ms·
One great problem - and I am but a hobbyist at best - is that of back-propagation, which is the foundation of deep learning neural networks. As far as I know o
by cr0sh 8y ago
One great problem - and I am but a hobbyist at best - is that of back-propagation, which is the foundation of deep learning neural networks.
As far as I know of (and I could be wrong) - there has yet to be any kind of proven method that the human brain uses any form of back-propagation for learning. From what I understand, there isn't a biological equivalent.
Which may be ok, in the long run - after all, we fly using fixed wings and propellers, and not flapping wings (as was tried over and over in the history of flight, with little to no progress and numerous failures). That is, success can be had by not replicating nature exactly.
ANNs do this at one level - neurons in the network are simplified mathematical representations. But the learning portion seems to add more complexity; the backpropagation algorithm is simple in concept, but seemingly more complex than what the human brain and natural neurons do for learning.
We also seem to learn things faster; natural neural networks - brains - regardless of species - only require relatively few examples to learn and generalize from, while deep learning artificial neural networks require thousands and more, of labeled examples.
These two issues - the need for backpropagation for learning, and the need for thousands and thousands of labeled examples exhaustively run through - both have resulted in machines and systems of hardware and software of immense complexity and power usage. We haven't even created the equivalent of a mouse brain, and if we did, it would take the energy output of Niagara Falls to run it.
From what I understand, this is also (in broad terms) Geoffrey Hinton's position as well; that our current systems, as well as they do perform (and do they - I'm still amazed at some of the things I saw my simplified implementation of NVidia's End-to-End model do to drive a simulated car around a track), are probably not the right implementation for going forward, and we should step back and find another way (Hinton's current thing is something called "Capsule Networks" which I haven't looked into too deeply - from what I have seen of them, they still appear overly complex, but again, that is just my hobbyist opinion, and certainly carries no weight here).
As a hobbyist in this field, I don't have any solutions to these issues, but they are something that keeps me thinking: How has nature actually done it? How does is it able to be done with so little energy input (beyond scale differences)? How does it work with so few training examples?
- mettamage 8y agoHobbyist here as well. > As far as I know of (and I could be wrong) - there has yet to be any kind of proven method that the human brain uses any form of back-propagation for learning. From what I understand, there isn't a biological equivalent. Is there research done on how individual or a small group of neurons learn? > We also seem to learn things faster; natural neural networks - brains - regardless of species - only require relatively few examples to learn and generalize from, while deep learning artificial neural networks require thousands and more, of labeled examples. When you take the point of your parent's post, then I'd assume having multiple neural nets learning at the same time and seeing things from different perspectives then we might get there as well. > We haven't even created the equivalent of a mouse brain, and if we did, it would take the energy output of Niagara Falls to run it. Our brain is specialized architecture. Emulating a brain on a Von Neumann machine designed computer is similar to a brain trying to emulate a computer. To me it makes sense that you'd need specialized hardware for this. I wonder if we could simulate a neural net of a simple creature, like a worm.
- visarga 8y ago> neurons in the network are simplified mathematical representations. That is the important difference. A biological neuron is more like a cell in the cell automaton (Conway's game of life). It is a dynamic process, even if the dynamics change by a small factor, it could have a very different behaviour. Artificial neurons on the other hand are much more uniform and lack the time dimension. If you account for dynamics, then learning in the brain is a completely different thing than learning in neural nets. It has been proven that Hebbian learning (local plasticity of synapses) approximates gradient based learning, though in simulation it attains lower accuracies. Another observation was that biological neural nets don't have a mechanism to communicate gradients back. But a deep learning experiment proved you can have fixed random back connections and the network will still learn. They don't actually need to be symmetrical connections. The real mystery is about loss functions. Such a loss function would have to be computed in a distributed fashion in the brain. They are the origin of the learning signal. The brain has many such loss functions that have been discovered by evolution. We have not understood loss functions well enough, and understanding them would be a huge leap forward. Another important observation is that the brain learns from the environment, not a fixed training set. Environments are richer sources, and our current crop of AI agents are playing all sorts of games, but they never have the same complexity as real world. Btw, tomorrow DeepMind will make a Starcraft demo.
- scottlocklin 8y agoI keep saying this to neural net enthusiasts: brains and neural nets don't have anything in common other than some nouns. Not network topology, not even level encoding (bio neural nets are pulse rate encoded -closest "neural net" to this is something like a liquid state machine, which is not easily simulated on a Von Neumann computer). We actually know an awful lot about how things in the visual cortex work. Nothing like a "deep network."
- lucidrains 8y agoWhich makes the compelling results of deep networks even more mysterious. (https://gandissect.csail.mit.edu/ https://gandissect.csail.mit.edu/) How could we have captured the seemingly enormously complex act of visual object recognition, with all the neurons and synapses and chemicals, in something so crude and simple as backpropagation and stochastic gradient descent with a bunch of ones and zeros? Or perhaps both processes, biological and artificial, point to a more fundamental emergent phenomenon of the computational universe we inhabit?
- scottlocklin 8y agoI'm pretty sure if you threw the number of grad students and GPUs at PGMs or various ensemble techniques, the results wouldn't be very different. Fundamentally prediction as we do it is just a form of data compression. I don't think brains work in a way we understand at all (though pieces like visual cortex are well understood).
- deleted 8y ago[deleted]