4 ms·

I tend to agree that evolution reaches a local optimum given enough time, but it seems that this chip is geared towards machine learning rather than biological

by jbarrow 12y ago

I tend to agree that evolution reaches a local optimum given enough time, but it seems that this chip is geared towards machine learning rather than biological accuracy. And currently integrate-and-fire spiking neurons don't appear to work better on the data we're interested in. In this light, and although CNNs aren't the only architecture, his criticisms may be a little more reasonable.

oldmanLecun 12y ago
From my understanding, the chip is geared towards implementing neural networks with low power consumption, which makes using spikes a reasonable design choice. So one can argue that using spikes is not about biological accuracy, but power efficiency. That is great that people want general purpose machine learning chips, the question is how to do it in low power. My guess is that the right architecture will be a mix of ML primitives mixed in with things like spikes (and perhaps other primitives seen found in biology).
- mjn 12y ago
  Yes, that's also my understanding. Important background is that this is not (or at least not solely) a commercial initiative by IBM to produce a machine-learning chip, though I'm sure they would love to sell some too. It's a DARPA initiative to find a way to greatly reduce the power budget needed for large-scale data-processing. And one of the starting hypotheses of this particular program, SyNAPSE, is that sparseness in time, aka spikiness, is part of why biological organisms seem capable of processing large amounts of video/etc. data with lower power budgets than computers seem to require. Here's an excerpt from their program statement [1]: Current computers are limited by the amount of power required to process large volumes of data. In contrast, biological neural systems, such as the brain, process large volumes of information in complex ways while consuming very little power. Power savings are achieved in neural systems by the sparse utilizations of hardware resources in time and space. Since many real-world problems are power limited and must process large volumes of data, neuromorphic computers have significant promise. That may or may not be a good hypothesis, but it seems interesting to investigate. In any case, LeCun's real beef is with the DARPA program managers: he thinks a different area of ANN research would've been a better allocation of funds, because in his view this is not among the most promising lines of research. Not an uncommon reaction to DARPA choices, and not always wrong either, but DARPA's got the money. [1] http://www.darpa.mil/Our_Work/DSO/Programs/Systems_of_Neuromorphic_Adaptive_Plastic_Scalable_Electronics_%28SYNAPSE%29.aspx http://www.darpa.mil/Our_Work/DSO/Programs/Systems_of_Neurom...

by jbarrow 12y ago

oldmanLecun 12y ago

From my understanding, the chip is geared towards implementing neural networks with low power consumption, which makes using spikes a reasonable design choice. So one can argue that using spikes is not about biological accuracy, but power efficiency. That is great that people want general purpose machine learning chips, the question is how to do it in low power. My guess is that the right architecture will be a mix of ML primitives mixed in with things like spikes (and perhaps other primitives seen found in biology).

mjn 12y ago
Yes, that's also my understanding. Important background is that this is not (or at least not solely) a commercial initiative by IBM to produce a machine-learning chip, though I'm sure they would love to sell some too. It's a DARPA initiative to find a way to greatly reduce the power budget needed for large-scale data-processing. And one of the starting hypotheses of this particular program, SyNAPSE, is that sparseness in time, aka spikiness, is part of why biological organisms seem capable of processing large amounts of video/etc. data with lower power budgets than computers seem to require. Here's an excerpt from their program statement [1]: Current computers are limited by the amount of power required to process large volumes of data. In contrast, biological neural systems, such as the brain, process large volumes of information in complex ways while consuming very little power. Power savings are achieved in neural systems by the sparse utilizations of hardware resources in time and space. Since many real-world problems are power limited and must process large volumes of data, neuromorphic computers have significant promise. That may or may not be a good hypothesis, but it seems interesting to investigate. In any case, LeCun's real beef is with the DARPA program managers: he thinks a different area of ANN research would've been a better allocation of funds, because in his view this is not among the most promising lines of research. Not an uncommon reaction to DARPA choices, and not always wrong either, but DARPA's got the money. [1] http://www.darpa.mil/Our_Work/DSO/Programs/Systems_of_Neuromorphic_Adaptive_Plastic_Scalable_Electronics_%28SYNAPSE%29.aspx http://www.darpa.mil/Our_Work/DSO/Programs/Systems_of_Neurom...