8 ms·

The Future of Neuromorphic Computing

55 points by anthotny 10y ago

startupdiscuss 10y ago
If these articles get into the math behind it, I think they will realize that, currently, the brain is just a metaphor for a style of computation. The article does state this towards the end: "Given the utter lack of consensus on how the brain actually works, these designs are more or less cartoons of what neuroscientists think might be happening." We don't really know how the brain does what it does.
return0 10y ago
I see nothing in these "neuromorphic" architectures than hogwash trying to bullshit governments into giving them money. There's no conceptual advancement offered by these computers that can't be simulated with matlab. Until the day when we actually learn how neurons work, these will just be extremely premature optimizations.
- deleted 10y ago
  [deleted]
- p1esk 10y ago
  These designs are advances in the field of computer architecture. They look at how brain processes information for ideas to make hardware more efficient, for some applications (such as pattern matching). Did you expect something more?
  - return0 10y ago
    They use very rudimentary sketches that have little to do with real neurons. ANNs have been mimicking these things in a slightly lower detail since the 60s. We can do better pattern matching with ANNs.
    - p1esk 10y ago
      I think you might be confused about terminology. Neuromorphic computing is running some known ANN model directly in hardware. Why do we want it? Because ANN models in software work well for pattern matching, and we want to speed it up/make it more efficient.
      - return0 10y ago
        Nope, ANN's and deep learning are not used by these boards (Neurogrid, zeroth, truenorth).
        p1esk 10y ago
        https://arxiv.org/abs/1603.08270 https://arxiv.org/abs/1603.08270 They have been designed, and are being used either for more efficient pattern matching, or to speed up brain simulations (again, using known neuronal models). You seem to expect something else from neuromorphic computing, why?
        return0 10y ago
        I stand with Yann Lecun's criticism on the article: https://m.facebook.com/yann.lecun/posts/10152184295832143 https://m.facebook.com/yann.lecun/posts/10152184295832143 > [the truenorth team had] to shoehorn a convnet on a chip that really wasn't designed for it. I mean, if the goal was to run a convnet at low power, they should have built a chip for that. The performance (and the accuracy) would be a lot better than this. They used their 'neuromorphic' chip in an explicitly non-neuromorphic way, basically approximately mapping deep learning processes to their chip. There is very little neuromorphicity (brain-likeness) about it (plasticity rules out of their ass, for starters). And they still get less than state-of-the art performance in most tasks! I expect 'neuromorphic' to be used when sound neuroscience is used in large scale implementations that allow us to actually simulate parts of the brain. Anything else, we call it what it is, ANNs.
        p1esk 10y ago
pron 10y ago
> the recent success of A.I. I guess they mean the recent success mostly due to modern hardware of 1960s statistical clustering and classification algorithms that for PR and historical purposes some people call "AI", but are currently unknown to have any significant relationship with what we call intelligence. When we achieve the capabilities of an insect we would be able to call our algorithms "AI" without getting red in the face, as we'd know there's a decent chance we're at least on the path to intelligence. Until then, let's just call them statistical learning. That wouldn't make them any less valuable, but would represent them much more realistically and fairly. It's funny how how statistics was once considered the worst kind of lie, and now for some it's becoming synonymous with intelligence.
jjaredsimpson 10y ago
I never understand the odd advantage that brains are assumed to have over machines when comparing power consumption. >... AlphaGo ... was able to beat a world-champion human player of Go, but only after it had trained ... running on approximately a million watts. (Its opponent’s brain, by contrast, would have been about fifty thousand times more energy-thrifty, consuming twenty watts.) A human brain has a severe limitation though. It can't consume more or less energy even if it I wanted to. AlphaGo could double, triple, etc its power consumption and expect to improve its performance. The brain also took decades to train. Computers also have the advantage of being identical. You can't train any brain to be a master level Go player. I just don't see brains as the high watermark of intelligence. They occupy a very specific niche in what I assume is a vast unbounded landscape of possible intelligences.
- pron 10y ago
  > The brain also took decades to train. The brain of an insect doesn't take decades to train, and we're currently unable to match its capabilities, either. > I just don't see brains as the high watermark of intelligence. They occupy a very specific niche in what I assume is a vast unbounded landscape of possible intelligences. That is a hypothetical claim because we don't know what intelligence is. Surely, some algorithms are much better at some tasks than the human brain, but that has been the case since the advent of computing, and it does not make them intelligent. Intelligence, or how we would currently define it colloquially and imprecisely, is an algorithm or a class of algorithms with some specific capabilities. Could those capabilities be taken further than the human brain? We certainly can't say that they cannot, but it's not obvious that they can, either. The only kind of intelligence we know, our own, comes with a host of disadvantages that may be features of the particular algorithm employed by the brain and/or to limitations of the hardware, but they could possibly be essential to intelligence itself. Who knows, maybe an intelligence with access to more powerful hardware would be more prone to incapacitating boredom and depression or other kinds of mental illness. This is just one hypothetical possibility, but given how limited our understanding of intelligence is, there are plenty of possible roadblocks ahead. Even if a higher intelligence than humans' is possible, its hypothetical achievements are uncertain. Some of the greatest problems encountered by humans are not constrained by intelligence but by resources and observations, and others (e.g. politics) are limited by powers of persuasion (that also don't seem to be simply correlated with intelligence). For example, what's limiting theoretical physics isn't brains but access to experiments, and what's limiting certain optimization problems are computational limits, for which our own intelligence, at least, does not give good approximate solutions at all.
deleted 10y ago
[deleted]
JackFr 10y ago
"Neuromorphic" = "Ornithopter of the mind" Giving up on flapping wings was the first step to flight.
- varjag 10y ago
  Indeed. Imagine that Wright Flyer never happened, but some time in 1940s, the progress in engines' specific thrust made a wing flapping machine able to take off. That's where we are with machine learning.
lend000 10y ago
There's a lot of backlash and/or dismissiveness on HN every time someone brings up neuromorphic architectures, and I think it has a lot to do with the same defensiveness that people display when their political beliefs are challenged. When neuromorphic architectures start bearing fruit, programmers will no longer be so in-demand for configuring the machines, as it will shift the balance of power towards hardware engineers and hard scientists.
- return0 10y ago
  Computational neuroscientists have been using simplified models like these for decades, and in principle the operation of these 'neuromorphic' neurons can already be simulated in large numbers in 'ordinary' computers. So, it's not clear at all what is to be gained. AFAIK, most of the neuroscience community considers Truenorth a marketing ploy. I don't think programmers should wait for these chips before they panic. They should already panic now, because deep learning works.
partycoder 10y ago
In the movie Terminator 2, a futuristic robot with advanced AI was developed by reverse engineering a futuristic chip. In reality, we do not need to reverse engineer a chip. We can just reverse engineer our own brains.
deepnotderp 10y ago
It bugs me when people always talk about "neuromorphic computing" and explore crazy ideas that never work and look at them in awe, but when anyone brings up a somewhat novel architecture for deep learning (nets that are being used today, successfully...) people say "that'll never work". For example, our startup uses analog computing to achieve accuracy roughly equivalent to digital circuits, yet we're told that we're crazy? Meanwhile people dreaming about memristors are showered with grants and money....
- andai 10y ago
  That sounds fascinating! Electrical, mechanical, hydraulic?
  - deepnotderp 10y ago
    What do you mean?
    - andai 10y ago
      I was asking what you meant! :) I've only heard of analog computers in an ancient context, so I have no idea what kinds people are working on nowadays. You mentioned "chip" in another comment, so I'm guessing it's not mechanical/hydraulic.
      - adornedCupcake 10y ago
        Analog as in analog electrical signals, almost surely.
- petra 10y ago
  You're from Isocline, right ? Your GPS chip was really good. But your SIMD chip will be much more impressive, right?
  - Quanticles 10y ago
    No, they are not from Isocline... There are groups at UCSB and U-Tenn working on analog neural network technologies as well.
bcatanzaro 10y ago
The reason AI has been so successful recently is that the research community has assumed a ruthlessly empirical philosophy: no idea, no matter how beautiful or interesting, is considered truly useful until it bears measurable results on some dataset. The reason neuromorphic computing gets such skepticism from AI researchers is that so far it has resisted any attempts at this kind of empiricism. No neuromorphic implementation has shown state of the art results on any important problem. If/When neuromorphic computers show groundbreaking results, the community will pivot quickly to using them. But expecting AI researchers to show deference to neuromorphic computing because it "mimics the brain" is to ignore the empirical philosophy that has led to AI's success.
- andreyk 10y ago
  To be fair, this whole Deep Learning renaissance was made possible and kicked off only after decades of research in multi layer neural nets (going back to the 80s) by Hinton, Lecun, etc. They stuck to their chosen method despite it not having great empirical results (the research community shunned NNs in the 90s for SVNs cause they worked better), because they believed it should and will work - and it did, eventually. So a similar argument for 'basic research' could be made for neuromorphic computing.
  - bcatanzaro 10y ago
    Yes, I totally agree. Yann LeCun, Geoff Hinton, Jurgen Schmidhuber and others did unpopular work for a long time. And they deserve tons of credit for their perseverance which paid off. Similarly, I think it's great that there are AI researchers working on techniques which are currently out of favor. It's important to have diversity of viewpoint. What irritates me about neuromorphic computing is that much of the work I see publicized (including the work in this article) isn't being presented as basic research on a risky hypothesis. Instead it's presented as the future of AI, despite the current lack of any demonstrated utility, and the almost complete disconnect between the AI researchers building the future of AI and the neuromorphic community. The burden of proof is always on the researcher to show utility, and if the neuromorphic computing community can do that, I'll be super excited! Until then, I'll be waiting for something measurable and concrete, and rolling my eyes at brain analogies.

55 points by anthotny 10y ago

If these articles get into the math behind it, I think they will realize that, currently, the brain is just a metaphor for a style of computation. The article does state this towards the end: "Given the utter lack of consensus on how the brain actually works, these designs are more or less cartoons of what neuroscientists think might be happening." We don't really know how the brain does what it does.

return0 10y ago

I see nothing in these "neuromorphic" architectures than hogwash trying to bullshit governments into giving them money. There's no conceptual advancement offered by these computers that can't be simulated with matlab. Until the day when we actually learn how neurons work, these will just be extremely premature optimizations.

deleted 10y ago
[deleted]
p1esk 10y ago
These designs are advances in the field of computer architecture. They look at how brain processes information for ideas to make hardware more efficient, for some applications (such as pattern matching). Did you expect something more?
- return0 10y ago
  They use very rudimentary sketches that have little to do with real neurons. ANNs have been mimicking these things in a slightly lower detail since the 60s. We can do better pattern matching with ANNs.
  - p1esk 10y ago
    I think you might be confused about terminology. Neuromorphic computing is running some known ANN model directly in hardware. Why do we want it? Because ANN models in software work well for pattern matching, and we want to speed it up/make it more efficient.
    - return0 10y ago
      Nope, ANN's and deep learning are not used by these boards (Neurogrid, zeroth, truenorth).
      - p1esk 10y ago
        https://arxiv.org/abs/1603.08270 https://arxiv.org/abs/1603.08270 They have been designed, and are being used either for more efficient pattern matching, or to speed up brain simulations (again, using known neuronal models). You seem to expect something else from neuromorphic computing, why?
        return0 10y ago
        I stand with Yann Lecun's criticism on the article: https://m.facebook.com/yann.lecun/posts/10152184295832143 https://m.facebook.com/yann.lecun/posts/10152184295832143 > [the truenorth team had] to shoehorn a convnet on a chip that really wasn't designed for it. I mean, if the goal was to run a convnet at low power, they should have built a chip for that. The performance (and the accuracy) would be a lot better than this. They used their 'neuromorphic' chip in an explicitly non-neuromorphic way, basically approximately mapping deep learning processes to their chip. There is very little neuromorphicity (brain-likeness) about it (plasticity rules out of their ass, for starters). And they still get less than state-of-the art performance in most tasks! I expect 'neuromorphic' to be used when sound neuroscience is used in large scale implementations that allow us to actually simulate parts of the brain. Anything else, we call it what it is, ANNs.
        p1esk 10y ago

pron 10y ago

> the recent success of A.I. I guess they mean the recent success mostly due to modern hardware of 1960s statistical clustering and classification algorithms that for PR and historical purposes some people call "AI", but are currently unknown to have any significant relationship with what we call intelligence. When we achieve the capabilities of an insect we would be able to call our algorithms "AI" without getting red in the face, as we'd know there's a decent chance we're at least on the path to intelligence. Until then, let's just call them statistical learning. That wouldn't make them any less valuable, but would represent them much more realistically and fairly. It's funny how how statistics was once considered the worst kind of lie, and now for some it's becoming synonymous with intelligence.

jjaredsimpson 10y ago

I never understand the odd advantage that brains are assumed to have over machines when comparing power consumption. >... AlphaGo ... was able to beat a world-champion human player of Go, but only after it had trained ... running on approximately a million watts. (Its opponent’s brain, by contrast, would have been about fifty thousand times more energy-thrifty, consuming twenty watts.) A human brain has a severe limitation though. It can't consume more or less energy even if it I wanted to. AlphaGo could double, triple, etc its power consumption and expect to improve its performance. The brain also took decades to train. Computers also have the advantage of being identical. You can't train any brain to be a master level Go player. I just don't see brains as the high watermark of intelligence. They occupy a very specific niche in what I assume is a vast unbounded landscape of possible intelligences.

pron 10y ago
> The brain also took decades to train. The brain of an insect doesn't take decades to train, and we're currently unable to match its capabilities, either. > I just don't see brains as the high watermark of intelligence. They occupy a very specific niche in what I assume is a vast unbounded landscape of possible intelligences. That is a hypothetical claim because we don't know what intelligence is. Surely, some algorithms are much better at some tasks than the human brain, but that has been the case since the advent of computing, and it does not make them intelligent. Intelligence, or how we would currently define it colloquially and imprecisely, is an algorithm or a class of algorithms with some specific capabilities. Could those capabilities be taken further than the human brain? We certainly can't say that they cannot, but it's not obvious that they can, either. The only kind of intelligence we know, our own, comes with a host of disadvantages that may be features of the particular algorithm employed by the brain and/or to limitations of the hardware, but they could possibly be essential to intelligence itself. Who knows, maybe an intelligence with access to more powerful hardware would be more prone to incapacitating boredom and depression or other kinds of mental illness. This is just one hypothetical possibility, but given how limited our understanding of intelligence is, there are plenty of possible roadblocks ahead. Even if a higher intelligence than humans' is possible, its hypothetical achievements are uncertain. Some of the greatest problems encountered by humans are not constrained by intelligence but by resources and observations, and others (e.g. politics) are limited by powers of persuasion (that also don't seem to be simply correlated with intelligence). For example, what's limiting theoretical physics isn't brains but access to experiments, and what's limiting certain optimization problems are computational limits, for which our own intelligence, at least, does not give good approximate solutions at all.

deleted 10y ago

[deleted]

JackFr 10y ago

"Neuromorphic" = "Ornithopter of the mind" Giving up on flapping wings was the first step to flight.

varjag 10y ago
Indeed. Imagine that Wright Flyer never happened, but some time in 1940s, the progress in engines' specific thrust made a wing flapping machine able to take off. That's where we are with machine learning.

lend000 10y ago

There's a lot of backlash and/or dismissiveness on HN every time someone brings up neuromorphic architectures, and I think it has a lot to do with the same defensiveness that people display when their political beliefs are challenged. When neuromorphic architectures start bearing fruit, programmers will no longer be so in-demand for configuring the machines, as it will shift the balance of power towards hardware engineers and hard scientists.

return0 10y ago
Computational neuroscientists have been using simplified models like these for decades, and in principle the operation of these 'neuromorphic' neurons can already be simulated in large numbers in 'ordinary' computers. So, it's not clear at all what is to be gained. AFAIK, most of the neuroscience community considers Truenorth a marketing ploy. I don't think programmers should wait for these chips before they panic. They should already panic now, because deep learning works.

partycoder 10y ago

In the movie Terminator 2, a futuristic robot with advanced AI was developed by reverse engineering a futuristic chip. In reality, we do not need to reverse engineer a chip. We can just reverse engineer our own brains.

deepnotderp 10y ago

It bugs me when people always talk about "neuromorphic computing" and explore crazy ideas that never work and look at them in awe, but when anyone brings up a somewhat novel architecture for deep learning (nets that are being used today, successfully...) people say "that'll never work". For example, our startup uses analog computing to achieve accuracy roughly equivalent to digital circuits, yet we're told that we're crazy? Meanwhile people dreaming about memristors are showered with grants and money....

andai 10y ago
That sounds fascinating! Electrical, mechanical, hydraulic?
- deepnotderp 10y ago
  What do you mean?
  - andai 10y ago
    I was asking what you meant! :) I've only heard of analog computers in an ancient context, so I have no idea what kinds people are working on nowadays. You mentioned "chip" in another comment, so I'm guessing it's not mechanical/hydraulic.
    - adornedCupcake 10y ago
      Analog as in analog electrical signals, almost surely.
petra 10y ago
You're from Isocline, right ? Your GPS chip was really good. But your SIMD chip will be much more impressive, right?
- Quanticles 10y ago
  No, they are not from Isocline... There are groups at UCSB and U-Tenn working on analog neural network technologies as well.

bcatanzaro 10y ago

The reason AI has been so successful recently is that the research community has assumed a ruthlessly empirical philosophy: no idea, no matter how beautiful or interesting, is considered truly useful until it bears measurable results on some dataset. The reason neuromorphic computing gets such skepticism from AI researchers is that so far it has resisted any attempts at this kind of empiricism. No neuromorphic implementation has shown state of the art results on any important problem. If/When neuromorphic computers show groundbreaking results, the community will pivot quickly to using them. But expecting AI researchers to show deference to neuromorphic computing because it "mimics the brain" is to ignore the empirical philosophy that has led to AI's success.

andreyk 10y ago
To be fair, this whole Deep Learning renaissance was made possible and kicked off only after decades of research in multi layer neural nets (going back to the 80s) by Hinton, Lecun, etc. They stuck to their chosen method despite it not having great empirical results (the research community shunned NNs in the 90s for SVNs cause they worked better), because they believed it should and will work - and it did, eventually. So a similar argument for 'basic research' could be made for neuromorphic computing.
- bcatanzaro 10y ago
  Yes, I totally agree. Yann LeCun, Geoff Hinton, Jurgen Schmidhuber and others did unpopular work for a long time. And they deserve tons of credit for their perseverance which paid off. Similarly, I think it's great that there are AI researchers working on techniques which are currently out of favor. It's important to have diversity of viewpoint. What irritates me about neuromorphic computing is that much of the work I see publicized (including the work in this article) isn't being presented as basic research on a risky hypothesis. Instead it's presented as the future of AI, despite the current lack of any demonstrated utility, and the almost complete disconnect between the AI researchers building the future of AI and the neuromorphic community. The burden of proof is always on the researcher to show utility, and if the neuromorphic computing community can do that, I'll be super excited! Until then, I'll be waiting for something measurable and concrete, and rolling my eyes at brain analogies.