4 ms·

As someone with only basic idea of deep learning, I want to ask this question: If deep learning is mimicking how our brains work, wouldn't it be theoretically

As someone with only basic idea of deep learning, I want to ask this question: If deep learning is mimicking how our brains work, wouldn't it be theoretically as powerful as our brains? In terms of architecture and complexity of the model.

currymj 8y ago
it's not really mimicking how our brains work, although that was the initial inspiration, decades ago. There are still people trying to model the actual brain but they don't overlap much with the machine learning community anymore. the idea originally was that a neuron can be roughly modeled as something that sums up inputs from other neurons and then outputs some kind of threshold/activation of those inputs. this is probably sort of true of biological neurons although as living cells they are much more complicated than that. but lots of recent and highly successful techniques in deep learning have given up on even that very weak amount of biological imitation. One simple example: lots of techniques involve multiplying together the outputs of "neurons". There is no sensible biological interpretation of this operation. Neural networks are Turing-complete (assuming all the usual pedantic assumptions you need to make that statement meaningful) so in that sense they are as powerful as our brains.
- danmaz74 8y ago
  Being Turing complete makes them as powerful as any other digital computer, but it has little (personally I would say nothing) to do with our brain.
  - currymj 8y ago
    Why not? A human given enough time and an infinite tape can easily simulate a Turing machine. The other way is more controversial, but I don't think it's outrageous to speculate that a brain could be accurately simulated by a Turing machine.
    - danmaz74 8y ago
      My point is that we know for a fact that our brain doesn't work anything like a Turing machine. Can a Turing machine simulate a brain - as in, create intelligence like the one our brains expose? That's very much an open question, but you wrote as if it was an obvious conclusion.
      - currymj 8y ago
        It's a pretty common assumption that any physical process could be simulated by a Turing machine (given infinite time and space etc.). We could probably never build such a Turing machine. I didn't mean to make any stronger claims than that. As I mentioned before, this is a pretty pedantic way of looking at the world.
        danmaz74 8y ago
        > We could probably never build such a Turing machine. I didn't mean to make any stronger claims than that Fair enough. I got a different impression from reading your original comment :)
      - edanm 8y ago
        > My point is that we know for a fact that our brain doesn't work anything like a Turing machine. That's of course true, but then most computers aren't that similar to a classic Turing machine either. The idea of Turing machines is that they can simulate anything else that we mean when we say "computation". > Can a Turing machine simulate a brain - as in, create intelligence like the one our brains expose? Yes, it can. I mean, if it can't, that means there's some notion of "computability" that a Turing machine doesn't capture - this would violate the "Church-Turing" thesis. Such a thing is possible, of course, since we have no proof that a Turing machine is the limit of what we mean by "computable" - but it would be a completely revolutionary fact in Computer Science, and one that is very unlikely, considering how many years people have been trying to come up with alternative models of "computation" that all turn out to be equivalent to Turing machines. - The human brain isn't magic - it works according to the laws of physics like anything else. Either it is simulatable by a Turing machine, which would make the most sense, or it isn't, in which case we're missing something about how you can compute things, which would be surprising and amazing.
thomasahle 8y ago
It's only mimicking brains in a very narrow/buzzword sense. In practice a full brain simulation, with millions of different neuron types is not feasible.
partycoder 8y ago
Artificial neural networks, etc. are an abstraction that is inspired on a biological neural ensemble. With many differences... Neurons are families of cells that come in many shapes and most of the time get oversimplified for didactic purposes. The processes involved in neuroscience are also oversimplified most of the time. It is often said that dendrites are the input, axons the output, and that spikes are fired when a threshold is met... but that's not always true. The biological situation is full of special cases. Then you have the way the network is formed. The amount of neurons and layers in a deep neural networks are just hardcoded in advance. That's not the case in biology. Then you have volume: biological neural networks have many more neurons, and many more connections between those. For example each Purkinje cell (in the cerebellum) can have over 200k dentritic connections.
mikehollinger 8y ago
There’s no such thing as a free lunch. In the case of deep learning - we’ve figured out as an industry how to apply a particular model of linear algebra to solve really interesting problems. It does way better than you’d guess in particular spaces. Hotdog or not hotdog is a perfect example of this. I actually (for fun) pulled several thousand pictures of hotdogs from the MS COCO dataset and trained a model to do that. It did well - but what I’d actually created wasn’t a hotdog detector - is created a bun detector. I could prove this by giving it an image of a sandwich and the model would confidently score it as a hotdog. We aren’t mimicking how our brains work in any broad sense. At best it’s a very narrow definition. In this case - the larger the matrices - the more you can do. There’ll probably be a Moore’s law of AI at some point.
- wetpaws 8y ago
  Given how much BS people typically believe, I would say a problem of training data is universal and not something exclusive to neural networks :)
- Baeocystin 8y ago
  Sounds like a real example the apocryphal tank/shadow detector. https://www.gwern.net/Tanks https://www.gwern.net/Tanks
  - gwern 8y ago
    If you showed a human who'd never heard of hotdogs before thousands of photos of hotdogs, and every single one of them had buns, why do you think they would assume that a naked wiener would be the true definition of hotdog, rather than 'bun with meat/filling'?
    - Baeocystin 8y ago
      >and every single one of them had buns Why would you assume that? I did a cursory GIS and Bing search, and somewhere between 1-5% of the returned images were bunless wieners.
MAXPOOL 8y ago
Deep learning is not mimicking how our brains work. It's just brain inspired. It may mimic of how some primitive areas of brain work in very low level in very crude way. Take for example machine vision. ConvNET is inspired by the lowest levels of human visual cortex and there are analogies to be drawn, but no direct mimicry.
- dnautics 8y ago
  is that even true? I could be wrong, but I was under the impression that Convnet is inspired by basically, photoshop filters, and after it was created it was found that there are vaguely similar structures in the visual cortex.
  - sjg007 8y ago
    The inspiration is probably from the signal processing literature.
    - p1esk 8y ago
      Convnet was inspired by Fukushima's Neocognitron, which itself was inspired by Hubel and Wiesel work on visual cortex. Here's more on convnets vs brain: https://neurdiness.wordpress.com/2018/05/17/deep-convolutional-neural-networks-as-models-of-the-visual-system-qa/ https://neurdiness.wordpress.com/2018/05/17/deep-convolution...
- eli_gottlieb 8y ago
  I'm pretty sure convnets were inspired by the lowest levels of the cat visual cortex.
  - DanielleMolloy 8y ago
    ..which are not very different from the human visual cortex.
patall 8y ago
I addition to what other have said, even if DL can mimick what a brain (or rather neural system) does, nobody could really say if its a fly, sea urchin or monkey brain. Sure the last has the most neurons but thats certainly not all. And simple brains have existed for hundereds of millions of years, but really intelligent creatures remain the exception.

currymj 8y ago

it's not really mimicking how our brains work, although that was the initial inspiration, decades ago. There are still people trying to model the actual brain but they don't overlap much with the machine learning community anymore. the idea originally was that a neuron can be roughly modeled as something that sums up inputs from other neurons and then outputs some kind of threshold/activation of those inputs. this is probably sort of true of biological neurons although as living cells they are much more complicated than that. but lots of recent and highly successful techniques in deep learning have given up on even that very weak amount of biological imitation. One simple example: lots of techniques involve multiplying together the outputs of "neurons". There is no sensible biological interpretation of this operation. Neural networks are Turing-complete (assuming all the usual pedantic assumptions you need to make that statement meaningful) so in that sense they are as powerful as our brains.

danmaz74 8y ago
Being Turing complete makes them as powerful as any other digital computer, but it has little (personally I would say nothing) to do with our brain.
- currymj 8y ago
  Why not? A human given enough time and an infinite tape can easily simulate a Turing machine. The other way is more controversial, but I don't think it's outrageous to speculate that a brain could be accurately simulated by a Turing machine.
  - danmaz74 8y ago
    My point is that we know for a fact that our brain doesn't work anything like a Turing machine. Can a Turing machine simulate a brain - as in, create intelligence like the one our brains expose? That's very much an open question, but you wrote as if it was an obvious conclusion.
    - currymj 8y ago
      It's a pretty common assumption that any physical process could be simulated by a Turing machine (given infinite time and space etc.). We could probably never build such a Turing machine. I didn't mean to make any stronger claims than that. As I mentioned before, this is a pretty pedantic way of looking at the world.
      - danmaz74 8y ago
        > We could probably never build such a Turing machine. I didn't mean to make any stronger claims than that Fair enough. I got a different impression from reading your original comment :)
    - edanm 8y ago
      > My point is that we know for a fact that our brain doesn't work anything like a Turing machine. That's of course true, but then most computers aren't that similar to a classic Turing machine either. The idea of Turing machines is that they can simulate anything else that we mean when we say "computation". > Can a Turing machine simulate a brain - as in, create intelligence like the one our brains expose? Yes, it can. I mean, if it can't, that means there's some notion of "computability" that a Turing machine doesn't capture - this would violate the "Church-Turing" thesis. Such a thing is possible, of course, since we have no proof that a Turing machine is the limit of what we mean by "computable" - but it would be a completely revolutionary fact in Computer Science, and one that is very unlikely, considering how many years people have been trying to come up with alternative models of "computation" that all turn out to be equivalent to Turing machines. - The human brain isn't magic - it works according to the laws of physics like anything else. Either it is simulatable by a Turing machine, which would make the most sense, or it isn't, in which case we're missing something about how you can compute things, which would be surprising and amazing.

thomasahle 8y ago

It's only mimicking brains in a very narrow/buzzword sense. In practice a full brain simulation, with millions of different neuron types is not feasible.

partycoder 8y ago

Artificial neural networks, etc. are an abstraction that is inspired on a biological neural ensemble. With many differences... Neurons are families of cells that come in many shapes and most of the time get oversimplified for didactic purposes. The processes involved in neuroscience are also oversimplified most of the time. It is often said that dendrites are the input, axons the output, and that spikes are fired when a threshold is met... but that's not always true. The biological situation is full of special cases. Then you have the way the network is formed. The amount of neurons and layers in a deep neural networks are just hardcoded in advance. That's not the case in biology. Then you have volume: biological neural networks have many more neurons, and many more connections between those. For example each Purkinje cell (in the cerebellum) can have over 200k dentritic connections.

mikehollinger 8y ago

There’s no such thing as a free lunch. In the case of deep learning - we’ve figured out as an industry how to apply a particular model of linear algebra to solve really interesting problems. It does way better than you’d guess in particular spaces. Hotdog or not hotdog is a perfect example of this. I actually (for fun) pulled several thousand pictures of hotdogs from the MS COCO dataset and trained a model to do that. It did well - but what I’d actually created wasn’t a hotdog detector - is created a bun detector. I could prove this by giving it an image of a sandwich and the model would confidently score it as a hotdog. We aren’t mimicking how our brains work in any broad sense. At best it’s a very narrow definition. In this case - the larger the matrices - the more you can do. There’ll probably be a Moore’s law of AI at some point.

wetpaws 8y ago
Given how much BS people typically believe, I would say a problem of training data is universal and not something exclusive to neural networks :)
Baeocystin 8y ago
Sounds like a real example the apocryphal tank/shadow detector. https://www.gwern.net/Tanks https://www.gwern.net/Tanks
- gwern 8y ago
  If you showed a human who'd never heard of hotdogs before thousands of photos of hotdogs, and every single one of them had buns, why do you think they would assume that a naked wiener would be the true definition of hotdog, rather than 'bun with meat/filling'?
  - Baeocystin 8y ago
    >and every single one of them had buns Why would you assume that? I did a cursory GIS and Bing search, and somewhere between 1-5% of the returned images were bunless wieners.

MAXPOOL 8y ago

Deep learning is not mimicking how our brains work. It's just brain inspired. It may mimic of how some primitive areas of brain work in very low level in very crude way. Take for example machine vision. ConvNET is inspired by the lowest levels of human visual cortex and there are analogies to be drawn, but no direct mimicry.

dnautics 8y ago
is that even true? I could be wrong, but I was under the impression that Convnet is inspired by basically, photoshop filters, and after it was created it was found that there are vaguely similar structures in the visual cortex.
- sjg007 8y ago
  The inspiration is probably from the signal processing literature.
  - p1esk 8y ago
    Convnet was inspired by Fukushima's Neocognitron, which itself was inspired by Hubel and Wiesel work on visual cortex. Here's more on convnets vs brain: https://neurdiness.wordpress.com/2018/05/17/deep-convolutional-neural-networks-as-models-of-the-visual-system-qa/ https://neurdiness.wordpress.com/2018/05/17/deep-convolution...
eli_gottlieb 8y ago
I'm pretty sure convnets were inspired by the lowest levels of the cat visual cortex.
- DanielleMolloy 8y ago
  ..which are not very different from the human visual cortex.

patall 8y ago

I addition to what other have said, even if DL can mimick what a brain (or rather neural system) does, nobody could really say if its a fly, sea urchin or monkey brain. Sure the last has the most neurons but thats certainly not all. And simple brains have existed for hundereds of millions of years, but really intelligent creatures remain the exception.