7 ms·

Brain vs. Deep Learning (2015)

170 points by akkishore 9y ago

alanbernstein 9y ago
This looks like a fascinating article that I will have to come back to. I'm not sure how well it fits the blog's tagline, "Making deep learning accessible." ...
- baking 9y ago
  Maybe if you read the title of his blog as "Making deep learning understandable" and this blog post as "If you understand how deep learning really works (and not just treat it as a magic black box) and understand how the brain works you will realize we are nowhere near the singularity" it makes perfect sense.
Seanny123 9y ago
The math this guy performs is so questionable and his idea that Deep Learning is the pinnacle of biologically plausible cognitive modelling is incorrect. I write about this [in a blog post](https://medium.com/@seanaubin/deep-learning-is-almost-the-brain-3aaecd924f3d https://medium.com/@seanaubin/deep-learning-is-almost-the-br...). tl;dr doing pure Deep Learning (also known as connectionist) models of the brain limits your tools in a bad way. Using tools from Dynamicism and Symbolicism is better. As proof, check out Spaun, the world's largest functioning brain model. Note: I mostly just disagree with him philosophically, in terms of his reasoning because it's overlooking some evidence. Don't really have an option on his conclusion. Probably agree with him more than I disagree with him.
philipkglass 9y ago
Long, and fascinating. I can see why this made the front page. I oscillated between 20% strident disagreement and 60% strident agreement. (The rest -- no strong feelings, or I feel that the questions are too ill-formed to answer.) To pick one point of disagreement, “We do not need as much computational power as the brain has, because our algorithms are (will be) better than that of the brain.” I hope you can see after the descriptions in this blog post that this statement is rather arrogant. Machines are already better than brains at many cognitive tasks of practical interest. Believing that we'll continue to find "tricks" to allow computers to outperform brains on useful cognitive tasks, despite the brains' much greater complexity, seems like a perfectly sober and conservative prediction. If I had to advance my own pet reasons for discounting the likelihood of a technological singularity, here are my top two: 1) It's a more challenging case of the general Fermi paradox. Show me the Hubble images of the computronium Dyson swarms. If it takes less than a century to go from the first transistorized computers to superintelligence, and superintelligence is as prone to run amok as Bostrom/Yudkowsky think, signs should already be visible from Earth. 2) You need experiments to validate scientific models. Even if a machine-intelligence could think a billion times faster than a biological intelligence, it couldn't complete experiments a billion times faster. Technologies that act on the material world will improve sublinearly with respect to thinking/computing power, for at least this reason and probably others as well.
- deleted 9y ago
  [deleted]
- darkmighty 9y ago
  I agree with you. Our current results are impressive enough for me to believe we're not that far from AGI (of the kind that would satisfy the author here). This article is already largely outdated because of advances in Reinforcement Learning and other Deep Learning applications that show very strong primordial features of what he claims as epitomes of human capability. For example, trained RL agents will recognize objects without being able to label them, having internal representations for position, object "type", functionality and relevance. You just have to train the agent with a problem that requires this kind of comprehension for efficient performance (kind of comprehension which isn't needed in simple image labeling). AlphaGo and even labeling are indeed milestones in superhuman performance, and I believe Terence Tao's view that AI is a moving target is relevant here. --- 1) Regarding the Fermi paradox, I haven't read too much technical or scientific work on it (it's highly speculative anyway), but I find Isaac Arthur's videos lay the basic ideas pretty well: https://www.youtube.com/watch?v=oIva_60l3ww&t=1016s https://www.youtube.com/watch?v=oIva_60l3ww&t=1016s It becomes pretty convincing that technological intelligent life is an incredible coincidence. There might not be any huge "filters", as they are called (stages which reduce the probability of observing TIL), such as a superintelligence extinction event, but an enormous sequence of minor filters, ranging from low probability of a planet with adequate conditions, to low probability of actual technological development, to low probability of occurrence in our light cone (we can only see fairly young distant galaxies), and limitations to the visibility and spreading velocity of such civilizations. There is a wide range of parameters such that this does not contradict the generalized Mediocrity principle, such that there is probably more than one TIL in the universe, but they're few and far from each other in space and time. 2) I don't follow. This objection is only valid for discovering new laws of physics -- technological development can happen even with fixed knowledge of basic physical laws. At this stage it's not clear that even continued refinement of physical models. For example, the discovery of quarks certainly helped understanding nature, but it hasn't led to any direct technological applications due to quark confinement, and the fact that the particles are just too small. Neutrinos are another conceptually important discovery that doesn't really have applications due to low interactivity with matter.
yters 9y ago
If intelligence is not computable then by definition deep learning is not capable of human intelligence. There are any reasons to think intelligence is beyond computation: Gödel's incompleteness theorems, no free lunch theorem, data processing inequality, Solomonoff induction and Kolmorov complexity is uncomputable, the halting problem, if the mind is code which program are you, all programs are finite yet we can think about infinity, split brain but unified consciousness, the inherent difference between third person and first person descriptions, reasoning about paradoxes, the ability to know we are wrong, the ability to write AI programs, the whole connectionist vs. modules problem Fodor points out, and probably many more.
- felippee 9y ago
  I think the major problem is to think of "intelligence" as a problem in "computation". We are so used to this framing, that it may seem foreign. Others will argue, that in principle it is a problem of computability but in extreme almost every problem is, but that is not necessarily how we frame other problems. I'd say "intelligence" is a control problem (as in controlling a robot). This framing, though subtle, makes the entire problem quite different. You no longer talk about computability but you talk about survival in "high temperature thermal bath" (or otherwise called "physical reality"), full of unpredictability and dangerous stuff. When you frame it like this, it is clear we have not even began to address the problem properly, not to even mention solving it.
  - chanakya 9y ago
    Aren't other problems getting framed that way, too? Car driving is not inherently a problem in computation, but is becoming one. Robotic control (or other control problems) are certainly being solved by computation. Perhaps treating AI as a derivative of the control branch of computation could practically help speed up progress in some areas, but it shouldn't fundamentally change its nature.
    - felippee 9y ago
      Well this is exactly my point. A lot of problems are currently framed as computation problem. As much as it might be useful for some, I'd argue that we should be careful with this. We get all the great marvels of deep learning, but robots remain stupid as bricks over 30 years of Moore's law. To me this (Moravec's paradox) is a signal that we are doing something wrong, and typically we do things wrong when they are not framed properly.
spot 9y ago
tldr: 2037-2080 for "brainlike computers". sounds like a reasonable estimate to me. but who would say that is "nowhere near"? A few decades compared to billions of years of evolution to create life here on earth? a few decades compared to millions of years to go from monkeys to humanity? compared to thousands of years to go from prehistory to contemplating this question? a date that many of us will live to see? IMO this means we are on the verge.
- sporkologist 9y ago
  Since we're already at the level of coding genes, we are short-circuiting the normal process of evolution, which was a trial-and-error process. We are living in interesting times.
- icc97 9y ago
  Slightly more in detail was his approximation that brain could perform 10^21 FLOPS vs 10^15 currently. But yeah 2080 is within my daughter's lifetime. My 1950s house I bought will probably still be standing. I think it also coincides with Elon Musk's guess for when there'd be 1M people living on Mars.
ewjordan 9y ago
The key to the claim that other computational estimates are way off is essentially that there's a lot of data crunching happening within a single neuron, rather than it being something we can model as collecting a bunch of inputs and either firing or not. He's arguing that each neuron does a ton of internal computation that can itself be modeled as a (sometimes very large) convolutional network. I think in a sense this is well-established when looking at real neurons, but several times in the article he uses phrases like "shown to be important for information processing", and that's where I get off the boat a bit. When you're saying that it's so important for information processing that it warrants a 1000x or more increase in the computational power necessary to implement an algorithm, I think it's necessary to dig into what the actual work being done there is, not just that there's some non-trivial transformation. A lot of interesting and extremely tough to model fluid and chemical dynamics are in play when I drink too much water and have to pee, but that doesn't mean that we need to understand them to build a waste disposal system using pipes. In particular, does the within-neuron processing actively tune itself based on the data it processes to an extent on-par with inter-neuron connections (in which case the argument that it's fundamental to the learning process would hold a lot more weight), or is it mostly static? I think a lot of us consider "important for information processing" to mean "is a meaningfully dynamic parameter involved in a learning algorithm", rather than an accidental shmearing of data. I'd really love more info on what the actual processing that's happening is.
- Seanny123 9y ago
  That's basically what the entire field of computational/theoretical neuroscience is trying to figure out right now. What's being computed? What's the right level of abstraction? Basically, it depends on what you're modelling: https://www.ncbi.nlm.nih.gov/pubmed/24709593 https://www.ncbi.nlm.nih.gov/pubmed/24709593
- Lost_BiomedE 9y ago
  I worked in this field a bit at Baylor. There are those who want to model the 'hardware' down to every complexity and those who want to figure out the 'software' that emerges from the physical hardware. The groups communicate but are fairly divided. Does it tune itself or is static? It definitely tunes, but it can be meaningful to the algorithm or ignored.
mos_basik 9y ago
>Quantum tunneling will become relevant in 2016-2017 and has to be taken into account from there on. New materials and "insulated" circuits are required to make everything work from here on. He wrote this in 2015. Was he correct? I don't really know where to start researching something like this; is there anyone here familiar with the field who could comment on it?
- ivan_ah 9y ago
  I think he's referring to the general idea that transistor fabrication size shrinking cannot last forever. Since atoms have a fixed physical size, if the shrinking continues indefinitely, at some point the "wires" in the transistors will become zero-atoms thick (actually a few atoms thick). See this for explanations better than I could https://en.wikipedia.org/wiki/5_nanometer https://en.wikipedia.org/wiki/5_nanometer Note the "quantum" here is not a good thing (like the theoretical "quantum speedup" possible for certain computations on quantum computers), but a bad thing: imagine you want to send current down this wire, but the current is jumpy and often leaks out of the wire to a neighbouring wire thus causing errors in computations.
markan 9y ago
An interesting read, but the conclusion that "the singularity is nowhere near" was reached by assuming that only neural modeling could get us there, and that assumption wasn't defended well. (In fact it looks rather dubious, given all the quasi-intelligent things computers have achieved without copying neural dynamics.)

170 points by akkishore 9y ago

This looks like a fascinating article that I will have to come back to. I'm not sure how well it fits the blog's tagline, "Making deep learning accessible." ...

baking 9y ago
Maybe if you read the title of his blog as "Making deep learning understandable" and this blog post as "If you understand how deep learning really works (and not just treat it as a magic black box) and understand how the brain works you will realize we are nowhere near the singularity" it makes perfect sense.

Seanny123 9y ago

The math this guy performs is so questionable and his idea that Deep Learning is the pinnacle of biologically plausible cognitive modelling is incorrect. I write about this [in a blog post](https://medium.com/@seanaubin/deep-learning-is-almost-the-brain-3aaecd924f3d https://medium.com/@seanaubin/deep-learning-is-almost-the-br...). tl;dr doing pure Deep Learning (also known as connectionist) models of the brain limits your tools in a bad way. Using tools from Dynamicism and Symbolicism is better. As proof, check out Spaun, the world's largest functioning brain model. Note: I mostly just disagree with him philosophically, in terms of his reasoning because it's overlooking some evidence. Don't really have an option on his conclusion. Probably agree with him more than I disagree with him.

philipkglass 9y ago

Long, and fascinating. I can see why this made the front page. I oscillated between 20% strident disagreement and 60% strident agreement. (The rest -- no strong feelings, or I feel that the questions are too ill-formed to answer.) To pick one point of disagreement, “We do not need as much computational power as the brain has, because our algorithms are (will be) better than that of the brain.” I hope you can see after the descriptions in this blog post that this statement is rather arrogant. Machines are already better than brains at many cognitive tasks of practical interest. Believing that we'll continue to find "tricks" to allow computers to outperform brains on useful cognitive tasks, despite the brains' much greater complexity, seems like a perfectly sober and conservative prediction. If I had to advance my own pet reasons for discounting the likelihood of a technological singularity, here are my top two: 1) It's a more challenging case of the general Fermi paradox. Show me the Hubble images of the computronium Dyson swarms. If it takes less than a century to go from the first transistorized computers to superintelligence, and superintelligence is as prone to run amok as Bostrom/Yudkowsky think, signs should already be visible from Earth. 2) You need experiments to validate scientific models. Even if a machine-intelligence could think a billion times faster than a biological intelligence, it couldn't complete experiments a billion times faster. Technologies that act on the material world will improve sublinearly with respect to thinking/computing power, for at least this reason and probably others as well.

deleted 9y ago
[deleted]
darkmighty 9y ago
I agree with you. Our current results are impressive enough for me to believe we're not that far from AGI (of the kind that would satisfy the author here). This article is already largely outdated because of advances in Reinforcement Learning and other Deep Learning applications that show very strong primordial features of what he claims as epitomes of human capability. For example, trained RL agents will recognize objects without being able to label them, having internal representations for position, object "type", functionality and relevance. You just have to train the agent with a problem that requires this kind of comprehension for efficient performance (kind of comprehension which isn't needed in simple image labeling). AlphaGo and even labeling are indeed milestones in superhuman performance, and I believe Terence Tao's view that AI is a moving target is relevant here. --- 1) Regarding the Fermi paradox, I haven't read too much technical or scientific work on it (it's highly speculative anyway), but I find Isaac Arthur's videos lay the basic ideas pretty well: https://www.youtube.com/watch?v=oIva_60l3ww&t=1016s https://www.youtube.com/watch?v=oIva_60l3ww&t=1016s It becomes pretty convincing that technological intelligent life is an incredible coincidence. There might not be any huge "filters", as they are called (stages which reduce the probability of observing TIL), such as a superintelligence extinction event, but an enormous sequence of minor filters, ranging from low probability of a planet with adequate conditions, to low probability of actual technological development, to low probability of occurrence in our light cone (we can only see fairly young distant galaxies), and limitations to the visibility and spreading velocity of such civilizations. There is a wide range of parameters such that this does not contradict the generalized Mediocrity principle, such that there is probably more than one TIL in the universe, but they're few and far from each other in space and time. 2) I don't follow. This objection is only valid for discovering new laws of physics -- technological development can happen even with fixed knowledge of basic physical laws. At this stage it's not clear that even continued refinement of physical models. For example, the discovery of quarks certainly helped understanding nature, but it hasn't led to any direct technological applications due to quark confinement, and the fact that the particles are just too small. Neutrinos are another conceptually important discovery that doesn't really have applications due to low interactivity with matter.

yters 9y ago

If intelligence is not computable then by definition deep learning is not capable of human intelligence. There are any reasons to think intelligence is beyond computation: Gödel's incompleteness theorems, no free lunch theorem, data processing inequality, Solomonoff induction and Kolmorov complexity is uncomputable, the halting problem, if the mind is code which program are you, all programs are finite yet we can think about infinity, split brain but unified consciousness, the inherent difference between third person and first person descriptions, reasoning about paradoxes, the ability to know we are wrong, the ability to write AI programs, the whole connectionist vs. modules problem Fodor points out, and probably many more.

felippee 9y ago
I think the major problem is to think of "intelligence" as a problem in "computation". We are so used to this framing, that it may seem foreign. Others will argue, that in principle it is a problem of computability but in extreme almost every problem is, but that is not necessarily how we frame other problems. I'd say "intelligence" is a control problem (as in controlling a robot). This framing, though subtle, makes the entire problem quite different. You no longer talk about computability but you talk about survival in "high temperature thermal bath" (or otherwise called "physical reality"), full of unpredictability and dangerous stuff. When you frame it like this, it is clear we have not even began to address the problem properly, not to even mention solving it.
- chanakya 9y ago
  Aren't other problems getting framed that way, too? Car driving is not inherently a problem in computation, but is becoming one. Robotic control (or other control problems) are certainly being solved by computation. Perhaps treating AI as a derivative of the control branch of computation could practically help speed up progress in some areas, but it shouldn't fundamentally change its nature.
  - felippee 9y ago
    Well this is exactly my point. A lot of problems are currently framed as computation problem. As much as it might be useful for some, I'd argue that we should be careful with this. We get all the great marvels of deep learning, but robots remain stupid as bricks over 30 years of Moore's law. To me this (Moravec's paradox) is a signal that we are doing something wrong, and typically we do things wrong when they are not framed properly.

spot 9y ago

tldr: 2037-2080 for "brainlike computers". sounds like a reasonable estimate to me. but who would say that is "nowhere near"? A few decades compared to billions of years of evolution to create life here on earth? a few decades compared to millions of years to go from monkeys to humanity? compared to thousands of years to go from prehistory to contemplating this question? a date that many of us will live to see? IMO this means we are on the verge.

sporkologist 9y ago
Since we're already at the level of coding genes, we are short-circuiting the normal process of evolution, which was a trial-and-error process. We are living in interesting times.
icc97 9y ago
Slightly more in detail was his approximation that brain could perform 10^21 FLOPS vs 10^15 currently. But yeah 2080 is within my daughter's lifetime. My 1950s house I bought will probably still be standing. I think it also coincides with Elon Musk's guess for when there'd be 1M people living on Mars.

ewjordan 9y ago

The key to the claim that other computational estimates are way off is essentially that there's a lot of data crunching happening within a single neuron, rather than it being something we can model as collecting a bunch of inputs and either firing or not. He's arguing that each neuron does a ton of internal computation that can itself be modeled as a (sometimes very large) convolutional network. I think in a sense this is well-established when looking at real neurons, but several times in the article he uses phrases like "shown to be important for information processing", and that's where I get off the boat a bit. When you're saying that it's so important for information processing that it warrants a 1000x or more increase in the computational power necessary to implement an algorithm, I think it's necessary to dig into what the actual work being done there is, not just that there's some non-trivial transformation. A lot of interesting and extremely tough to model fluid and chemical dynamics are in play when I drink too much water and have to pee, but that doesn't mean that we need to understand them to build a waste disposal system using pipes. In particular, does the within-neuron processing actively tune itself based on the data it processes to an extent on-par with inter-neuron connections (in which case the argument that it's fundamental to the learning process would hold a lot more weight), or is it mostly static? I think a lot of us consider "important for information processing" to mean "is a meaningfully dynamic parameter involved in a learning algorithm", rather than an accidental shmearing of data. I'd really love more info on what the actual processing that's happening is.

Seanny123 9y ago
That's basically what the entire field of computational/theoretical neuroscience is trying to figure out right now. What's being computed? What's the right level of abstraction? Basically, it depends on what you're modelling: https://www.ncbi.nlm.nih.gov/pubmed/24709593 https://www.ncbi.nlm.nih.gov/pubmed/24709593
Lost_BiomedE 9y ago
I worked in this field a bit at Baylor. There are those who want to model the 'hardware' down to every complexity and those who want to figure out the 'software' that emerges from the physical hardware. The groups communicate but are fairly divided. Does it tune itself or is static? It definitely tunes, but it can be meaningful to the algorithm or ignored.

mos_basik 9y ago

>Quantum tunneling will become relevant in 2016-2017 and has to be taken into account from there on. New materials and "insulated" circuits are required to make everything work from here on. He wrote this in 2015. Was he correct? I don't really know where to start researching something like this; is there anyone here familiar with the field who could comment on it?

ivan_ah 9y ago
I think he's referring to the general idea that transistor fabrication size shrinking cannot last forever. Since atoms have a fixed physical size, if the shrinking continues indefinitely, at some point the "wires" in the transistors will become zero-atoms thick (actually a few atoms thick). See this for explanations better than I could https://en.wikipedia.org/wiki/5_nanometer https://en.wikipedia.org/wiki/5_nanometer Note the "quantum" here is not a good thing (like the theoretical "quantum speedup" possible for certain computations on quantum computers), but a bad thing: imagine you want to send current down this wire, but the current is jumpy and often leaks out of the wire to a neighbouring wire thus causing errors in computations.

markan 9y ago

An interesting read, but the conclusion that "the singularity is nowhere near" was reached by assuming that only neural modeling could get us there, and that assumption wasn't defended well. (In fact it looks rather dubious, given all the quasi-intelligent things computers have achieved without copying neural dynamics.)