4 ms·

How 'deep' are the networks used in something like DeepMask, and how does it compare with the number of layers of the human brain?

by achr2 10y ago

nickparker 10y ago
As I understand it, you can't meaningfully ask about the number of layers in the human brain. Layers are an abstraction we use to make artificial neural networks dramatically faster to work with using linear algebra. By keeping each node interacting only with the layers above and below, we make the computations a lot nicer for our model of computation. In the brain, neurons link freely to other arbitrary neurons based on adaption processes we (or at least I) don't really understand. The brain also lacks a clear idea of direction to count the layers along, since it has innumerable different inputs coming in at all times, and the resulting signals interact all over the place. The most meaningful analog would probably be to ask "How many neuron firings typically occur between an external stimulus and a response to that stimulus?" Even that is extremely rough though, because through evolution a lot of 'short circuit' structures have formed in our bodies. The gag reflex is obviously triggered by sensory input, but it probably doesn't check with your frontal lobe before firing the appropriate muscles.
dharma1 10y ago
The architecture of the brain is not really known and quite different to deep convolutional networks - but roughly 100 million neurons in a human brain, each one with thousands of synapses. It's several orders of magnitude more "neurons" and "connections" than even the largest ANN's
sushirain 10y ago
For example, ResNet from 2015 had 152 layers. A real neuron takes in the order of 10ms to integrate and fire to the next neuron. Many subconscious reactions take less than 1 sec, which leaves time to a chain of length less than 100. Note that those neurons are not strictly arranged in layers. The human visual cortex has 10^12 synapses [1]. One popular 2015 deep learning net (ResNet 152-layers) used 10^12 FLOPs to classify objects in one image (but less weights.) In terms of depth, we're there. In terms of breadth, it will take several years. But the brain does things very differently. For example, it has top-down signals during "prediction." [1] http://www.ncbi.nlm.nih.gov/pubmed/7244322 http://www.ncbi.nlm.nih.gov/pubmed/7244322
- tkinom 10y ago
  "It will take several years." A Noob question: If it reaches the "breadth" of human brain, how close will that to the "Skynet becomes self-aware" moment. What would it tell us about human, our society after it study, analyze millions, billions hours of FB, youtube videos?
  - visarga 10y ago
    As long as we do simple data processing, it won't become the dreaded Skynet. When we stuck a Reinforcement Learning system on top, feed it with reward signals and let it loose on the net or in real life robots, then it could become intelligent and develop its own goals/interests. So the Skynet part comes after reinforcement learning.
- visarga 10y ago
  It would seem that the current release from FB does top-down and bottom-up integration: > To capture general object shape, you have to have a high-level understanding of what you are looking at (DeepMask), but to accurately place the boundaries you need to look back at lower-level features all the way down to the pixels (SharpMask).

How 'deep' are the networks used in something like DeepMask, and how does it compare with the number of layers of the human brain?

by achr2 10y ago

How 'deep' are the networks used in something like DeepMask, and how does it compare with the number of layers of the human brain?

nickparker 10y ago

As I understand it, you can't meaningfully ask about the number of layers in the human brain. Layers are an abstraction we use to make artificial neural networks dramatically faster to work with using linear algebra. By keeping each node interacting only with the layers above and below, we make the computations a lot nicer for our model of computation. In the brain, neurons link freely to other arbitrary neurons based on adaption processes we (or at least I) don't really understand. The brain also lacks a clear idea of direction to count the layers along, since it has innumerable different inputs coming in at all times, and the resulting signals interact all over the place. The most meaningful analog would probably be to ask "How many neuron firings typically occur between an external stimulus and a response to that stimulus?" Even that is extremely rough though, because through evolution a lot of 'short circuit' structures have formed in our bodies. The gag reflex is obviously triggered by sensory input, but it probably doesn't check with your frontal lobe before firing the appropriate muscles.

dharma1 10y ago

The architecture of the brain is not really known and quite different to deep convolutional networks - but roughly 100 million neurons in a human brain, each one with thousands of synapses. It's several orders of magnitude more "neurons" and "connections" than even the largest ANN's

sushirain 10y ago

For example, ResNet from 2015 had 152 layers. A real neuron takes in the order of 10ms to integrate and fire to the next neuron. Many subconscious reactions take less than 1 sec, which leaves time to a chain of length less than 100. Note that those neurons are not strictly arranged in layers. The human visual cortex has 10^12 synapses [1]. One popular 2015 deep learning net (ResNet 152-layers) used 10^12 FLOPs to classify objects in one image (but less weights.) In terms of depth, we're there. In terms of breadth, it will take several years. But the brain does things very differently. For example, it has top-down signals during "prediction." [1] http://www.ncbi.nlm.nih.gov/pubmed/7244322 http://www.ncbi.nlm.nih.gov/pubmed/7244322

tkinom 10y ago
"It will take several years." A Noob question: If it reaches the "breadth" of human brain, how close will that to the "Skynet becomes self-aware" moment. What would it tell us about human, our society after it study, analyze millions, billions hours of FB, youtube videos?
- visarga 10y ago
  As long as we do simple data processing, it won't become the dreaded Skynet. When we stuck a Reinforcement Learning system on top, feed it with reward signals and let it loose on the net or in real life robots, then it could become intelligent and develop its own goals/interests. So the Skynet part comes after reinforcement learning.
visarga 10y ago
It would seem that the current release from FB does top-down and bottom-up integration: > To capture general object shape, you have to have a high-level understanding of what you are looking at (DeepMask), but to accurately place the boundaries you need to look back at lower-level features all the way down to the pixels (SharpMask).