5 ms·

>Deep learning, modeled on the human brain, is infinitely more complex [than machine learning]. Unlike machine learning, deep learning can teach machines to ign

by rm999 10y ago

>Deep learning, modeled on the human brain, is infinitely more complex [than machine learning]. Unlike machine learning, deep learning can teach machines to ignore all but the important characteristics of a sound or image-a hierarchical view of the world that accounts for infinite variety. -Yann LeCun I strongly disagree with a lot about this quote, even though it comes from a brilliant man I highly respect (my thesis research was inspired from some of his older work, and in my job we work with techniques he developed around deep learning). What I dislike is it utilizes hype to make deep learning seem mystical; in actuality it's a natural extension of old techniques that clearly fall under "machine learning". Deep learning is a neural network with 3 or more layers instead of the 2 layer networks that were developed in the 1980s. People tried 3 layers back then, and they didn't work well. Yann LeCunn and other researchers found cool ways to get 3+ layers to work in the 90s and again in the mid 2000s. More recently, researchers have just thrown a ton of data and computational power at them to get them to work. But fundamentally this was an extension of established techniques. This article that recently hit the front page actually breaks it down really well: https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/ https://blogs.nvidia.com/blog/2016/07/29/whats-difference-ar... I think my main point here is that deep learning is quite accessible to people who are learning machine learning. It's great at solving some really complex problems (that can certainly resemble true intelligence), but is not the right tool for other problems.

AnimalMuppet 10y ago
Viewed strictly as a neural network, how many layers deep is the human brain?
- soperj 10y ago
  I find that to be a very interesting question.
- kmicklas 10y ago
  I remember reading that the fastest human reflexes and computational outputs seem to happen around 100ms, and it takes roughly 10ms to transmit a signal from neuron to neuron, so that would imply 10 layers deep is enough for, for example, human level visual recognition. Of course, the human brain is highly recurrent, so it gets much more complicated for any higher level thinking. And it's also much, much wider than our neural networks, even if not spectacularly deep.
- quonn 10y ago
  https://en.wikipedia.org/wiki/Cortical_column https://en.wikipedia.org/wiki/Cortical_column https://en.wikipedia.org/wiki/Neocortex#Anatomy https://en.wikipedia.org/wiki/Neocortex#Anatomy
- conjectures 10y ago
  It contains feedback loops for a start, so the notion of depth may be misleading.
- cynusx 10y ago
  The human brain is what they call a recurrent neural network which means that it contains directed cycles. Classic neural networks are strictly hierarchical and one-way. If you take into account that signals can keep resonating in the brain (firing in circles) then the amount of layers is infinite. Recurrent neural networks are incredibly hard to train with the exception of liquid state machines (also known as echo state networks or reservoir computing).
- lossolo 10y ago
  Brain doesn't work like NNs in software. It's different beast you can't really compare layers in one and in second. There are several reason why this model just does not apply to the brain[1]: 1. The brain is not organized in layers this way. Instead there are interconnected circuits, feedback, brain regions, bidirectional connections, and such. 2.The brain's connections are feedforward only without "error feedback". 3. The brain doesn't learn from training data with correct answers (or a "ground truth"). 4.The neurons in the brain spike and use different learning algorithms. [1]https://www.quora.com/How-many-hidden-layers-are-in-the-neural-network-of-the-human-brain https://www.quora.com/How-many-hidden-layers-are-in-the-neur...
  - dmreedy 10y ago
    Just wanted to add that there _are_ certain portions of the brain which do seem to be layered, namely fundamental image-recognizers in the visual cortex. These were a lot of the inspiration for layered neural nets, and may explain some of their success on similar tasks.
    - JD557 10y ago
      I also believe that some[1] believe that CNNs and other image processing networks do some extra work that our brains don't need, as we can "cheat". For example, we get some translation-invariance for free with the ability to move our eyes. [1] http://ilp.mit.edu/event.jsp?id=6094 http://ilp.mit.edu/event.jsp?id=6094 (I had the video URL bookmarked, but it's broken now)
    - Cybiote 10y ago
      As I point out in another post (currently above) the layers of Neural nets have nothing in common with those in the brain. Only confusion will be met in heading down that path. What is had in common, and what was inspirational were (harsh) experiments on kittens, done in the 60s which provided hints as to what concepts were important in the primary visual cortex. These experiments lead to ideas of pooling and averaging as ways to calculate invariance, which made their way into the neurocognitron and then the convolutional net. The concepts of thresholding units and summation of inputs also played a part in the designs of Artificial Nets more generally. Most recently, new artificial NN designs have been loosely inspired by the importance of attention in biological networks. Another similarity, but this is a general computational similarity, is the importance of hierarchical organization in easing learning difficulty. But unlike in the mostly feedforward networks we're used to in ANNs, brain's networks are also feedback, or with backward projections, across areas and up and down hiearchies and so on. But there is more to vision than V1 and V4. The visual cortex has more arbitrary connections across layers and is not restricted to the bipartite structure common in MLPs and is more fine grained than the hidden layer summation across timesteps of RNNs. Other things the visual cortex computes but not well captured in models: speed and directional selectivity, nontrivial color transformations, temporal frequency, global v local + foreground v background, optical flow, motion tracking, inhibitory neurons, higher level scene understanding, feedback connections, perceptual color constancy and contributions to memory formation.
- jcranmer 10y ago
  Neural networks are based on a very, very outdated (and incorrect) model of how a neuron works, so your question is roughly akin to asking "how many vacuum tubes are in my GPU?" Mammalian brains are so densely packed that we don't really know what their internal architecture actually looks like--that's why estimates of the number of neurons keeps increasing over time [1]. Most of our models are based on some autopsies, and rely on things like fMRI or electroencephalography (not often you get to spell that word!) to work out which portions of the brain are involved in certain tasks. We have no hope of being able to build a connectivity graph between neurons at our current state of technology. The short answer is that the cerebral cortex is usually identified as having 6 layers. These can't be correlated with layers in a neural network, however. I do seem to recall a spitball number of 30-ish in relation to the visual cortex, but I've been unable to find any references as to what that number would actually represent, let alone if it's been superseded. [1] The current estimate is around 100 billion. If you want to know the number involved in, say, the visual cortex, that appears to be on the order of a few billion.
  - daveguy 10y ago
    > The short answer is that the cerebral cortex is usually identified as having 6 layers. These can't be correlated with layers in a neural network, however. I do seem to recall a spitball number of 30-ish in relation to the visual cortex, but I've been unable to find any references as to what that number would actually represent, let alone if it's been superseded. That short answer is very misleading. Those 6 layers are histological layers of density and structure in the cerebral cortex (eg how the cross sections look a little different). You might have many (30+) layers of cells with high interconnectivity in each of those 6 layers. Current estimate is 100 billion for the number of neurons, but the estimate of inter-connectivity is on the order of 100 trillion synapses. The human brain complexity many orders of magnitude higher than anything we have ever "simulated" in silicon.
    - Cybiote 10y ago
      It's not misleading because they were careful to point out that these 6 'layers' have nothing in common with how layers are defined in Neural networks. In Neural nets, layers are effectively subgraphs that are bipartite. They can also be computational stages in some networks like convolutional. The brain's layers are both much more loose in their allowed interconnections but also more modular in their functionality. There are different visual areas (usually 5) at different locations of the neocortex that connect to each other (feedforward and back) in complex ways. As residents of the neocortex, each visual area will thus have 6 layers which are in turn dividable into sublayers (often 1 - 4). These layers are structural but are also partially organizational tools for the scientists.
- chimtim 10y ago
  In terms of number of neurons, human brain is around 10^11. Current image classification deep networks are around 10^7. However, its not just the number of neurons, the connections per neurons also matter. The connections per neuron in a human are in thousands while in modern networks they are in hundreds. The human brain is also very efficient so it also makes a very good use of those neurons, and modern networks aren't. However, a human brain trains over a very very long time and also prunes itself constantly, while such efficiency techniques at scale for very large networks are being explored in deep learning only recently.
mathgenius 10y ago
> Deep learning, ... is infinitely more complex When I read this I interpret it to mean they have no idea how deep learning works. (More specifically: why it works as well as it does.) And the part about being modeled on the human brain just seems to be a fortunate (?) coincidence.
- albertop 10y ago
  You know who the Yann LeCun is, don't you?
- rm999 10y ago
  If you mean the lack of theoretical guarantees due to a highly non-convex optimization, you could argue the same thing about 30 year old 2-layer networks. And we actually do have a decent idea why deep learning works well: http://arxiv.org/pdf/1412.6544v4.pdf http://arxiv.org/pdf/1412.6544v4.pdf http://arxiv.org/pdf/1412.0233.pdf http://arxiv.org/pdf/1412.0233.pdf
  - mathgenius 10y ago
    Thankyou that looks awesome. Btw, "why" questions are always a bit ill-defined, it's like asking for some kind of narrative to satisfy the questioner. "well in the beginning there was the big-bang..." So anyway I'm quite impressed at these papers.

by rm999 10y ago

AnimalMuppet 10y ago

Viewed strictly as a neural network, how many layers deep is the human brain?

soperj 10y ago
I find that to be a very interesting question.
kmicklas 10y ago
I remember reading that the fastest human reflexes and computational outputs seem to happen around 100ms, and it takes roughly 10ms to transmit a signal from neuron to neuron, so that would imply 10 layers deep is enough for, for example, human level visual recognition. Of course, the human brain is highly recurrent, so it gets much more complicated for any higher level thinking. And it's also much, much wider than our neural networks, even if not spectacularly deep.
quonn 10y ago
https://en.wikipedia.org/wiki/Cortical_column https://en.wikipedia.org/wiki/Cortical_column https://en.wikipedia.org/wiki/Neocortex#Anatomy https://en.wikipedia.org/wiki/Neocortex#Anatomy
conjectures 10y ago
It contains feedback loops for a start, so the notion of depth may be misleading.
cynusx 10y ago
The human brain is what they call a recurrent neural network which means that it contains directed cycles. Classic neural networks are strictly hierarchical and one-way. If you take into account that signals can keep resonating in the brain (firing in circles) then the amount of layers is infinite. Recurrent neural networks are incredibly hard to train with the exception of liquid state machines (also known as echo state networks or reservoir computing).
lossolo 10y ago
Brain doesn't work like NNs in software. It's different beast you can't really compare layers in one and in second. There are several reason why this model just does not apply to the brain[1]: 1. The brain is not organized in layers this way. Instead there are interconnected circuits, feedback, brain regions, bidirectional connections, and such. 2.The brain's connections are feedforward only without "error feedback". 3. The brain doesn't learn from training data with correct answers (or a "ground truth"). 4.The neurons in the brain spike and use different learning algorithms. [1]https://www.quora.com/How-many-hidden-layers-are-in-the-neural-network-of-the-human-brain https://www.quora.com/How-many-hidden-layers-are-in-the-neur...
- dmreedy 10y ago
  Just wanted to add that there _are_ certain portions of the brain which do seem to be layered, namely fundamental image-recognizers in the visual cortex. These were a lot of the inspiration for layered neural nets, and may explain some of their success on similar tasks.
  - JD557 10y ago
    I also believe that some[1] believe that CNNs and other image processing networks do some extra work that our brains don't need, as we can "cheat". For example, we get some translation-invariance for free with the ability to move our eyes. [1] http://ilp.mit.edu/event.jsp?id=6094 http://ilp.mit.edu/event.jsp?id=6094 (I had the video URL bookmarked, but it's broken now)
  - Cybiote 10y ago
    As I point out in another post (currently above) the layers of Neural nets have nothing in common with those in the brain. Only confusion will be met in heading down that path. What is had in common, and what was inspirational were (harsh) experiments on kittens, done in the 60s which provided hints as to what concepts were important in the primary visual cortex. These experiments lead to ideas of pooling and averaging as ways to calculate invariance, which made their way into the neurocognitron and then the convolutional net. The concepts of thresholding units and summation of inputs also played a part in the designs of Artificial Nets more generally. Most recently, new artificial NN designs have been loosely inspired by the importance of attention in biological networks. Another similarity, but this is a general computational similarity, is the importance of hierarchical organization in easing learning difficulty. But unlike in the mostly feedforward networks we're used to in ANNs, brain's networks are also feedback, or with backward projections, across areas and up and down hiearchies and so on. But there is more to vision than V1 and V4. The visual cortex has more arbitrary connections across layers and is not restricted to the bipartite structure common in MLPs and is more fine grained than the hidden layer summation across timesteps of RNNs. Other things the visual cortex computes but not well captured in models: speed and directional selectivity, nontrivial color transformations, temporal frequency, global v local + foreground v background, optical flow, motion tracking, inhibitory neurons, higher level scene understanding, feedback connections, perceptual color constancy and contributions to memory formation.
jcranmer 10y ago
Neural networks are based on a very, very outdated (and incorrect) model of how a neuron works, so your question is roughly akin to asking "how many vacuum tubes are in my GPU?" Mammalian brains are so densely packed that we don't really know what their internal architecture actually looks like--that's why estimates of the number of neurons keeps increasing over time [1]. Most of our models are based on some autopsies, and rely on things like fMRI or electroencephalography (not often you get to spell that word!) to work out which portions of the brain are involved in certain tasks. We have no hope of being able to build a connectivity graph between neurons at our current state of technology. The short answer is that the cerebral cortex is usually identified as having 6 layers. These can't be correlated with layers in a neural network, however. I do seem to recall a spitball number of 30-ish in relation to the visual cortex, but I've been unable to find any references as to what that number would actually represent, let alone if it's been superseded. [1] The current estimate is around 100 billion. If you want to know the number involved in, say, the visual cortex, that appears to be on the order of a few billion.
- daveguy 10y ago
  > The short answer is that the cerebral cortex is usually identified as having 6 layers. These can't be correlated with layers in a neural network, however. I do seem to recall a spitball number of 30-ish in relation to the visual cortex, but I've been unable to find any references as to what that number would actually represent, let alone if it's been superseded. That short answer is very misleading. Those 6 layers are histological layers of density and structure in the cerebral cortex (eg how the cross sections look a little different). You might have many (30+) layers of cells with high interconnectivity in each of those 6 layers. Current estimate is 100 billion for the number of neurons, but the estimate of inter-connectivity is on the order of 100 trillion synapses. The human brain complexity many orders of magnitude higher than anything we have ever "simulated" in silicon.
  - Cybiote 10y ago
    It's not misleading because they were careful to point out that these 6 'layers' have nothing in common with how layers are defined in Neural networks. In Neural nets, layers are effectively subgraphs that are bipartite. They can also be computational stages in some networks like convolutional. The brain's layers are both much more loose in their allowed interconnections but also more modular in their functionality. There are different visual areas (usually 5) at different locations of the neocortex that connect to each other (feedforward and back) in complex ways. As residents of the neocortex, each visual area will thus have 6 layers which are in turn dividable into sublayers (often 1 - 4). These layers are structural but are also partially organizational tools for the scientists.
chimtim 10y ago
In terms of number of neurons, human brain is around 10^11. Current image classification deep networks are around 10^7. However, its not just the number of neurons, the connections per neurons also matter. The connections per neuron in a human are in thousands while in modern networks they are in hundreds. The human brain is also very efficient so it also makes a very good use of those neurons, and modern networks aren't. However, a human brain trains over a very very long time and also prunes itself constantly, while such efficiency techniques at scale for very large networks are being explored in deep learning only recently.

mathgenius 10y ago

> Deep learning, ... is infinitely more complex When I read this I interpret it to mean they have no idea how deep learning works. (More specifically: why it works as well as it does.) And the part about being modeled on the human brain just seems to be a fortunate (?) coincidence.

albertop 10y ago
You know who the Yann LeCun is, don't you?
rm999 10y ago
If you mean the lack of theoretical guarantees due to a highly non-convex optimization, you could argue the same thing about 30 year old 2-layer networks. And we actually do have a decent idea why deep learning works well: http://arxiv.org/pdf/1412.6544v4.pdf http://arxiv.org/pdf/1412.6544v4.pdf http://arxiv.org/pdf/1412.0233.pdf http://arxiv.org/pdf/1412.0233.pdf
- mathgenius 10y ago
  Thankyou that looks awesome. Btw, "why" questions are always a bit ill-defined, it's like asking for some kind of narrative to satisfy the questioner. "well in the beginning there was the big-bang..." So anyway I'm quite impressed at these papers.