6 ms·

Intel Gets Serious About Neuromorphic, Cognitive Computing Future

149 points by Katydid 10y ago

emcq 10y ago
The near future is not in putting learning inside a chip. We're a long ways off from the one shot learning needed to make a device actually interesting with localized learning. Instead, the future is recording and uploading your observations to the cloud, with data scientist and neural net wizards training over this dataset on a cluster with tons of GPUs, and then deploying an optimized model to scrappy low precision inference chips. This is why FPGA based designs will fail to be stunning. Specialized low precision ASICs more similar to DSPs like Movidius' Myriad (in the Phantom drones and Google's Project Tango devices), Google's TPU, upcoming Qualcomm chips, or Nervana's will become increasingly popular.
paulsutter 10y ago
The article misses the core issue: neural network architectures are still in flux. "Neuromorphic" chips are hardwired to one architecture, which makes them power efficient but less flexible. When designs are more stable, such chips could be more practical. Meanwhile, upcoming (non-neuromorphic) AI processors are taking two directions: larger numbers of simplified GPU-type cores (such as NVIDIA Xavier and Intel's Lake Crest/Nervana chips), and FPGAs. Simplifying cores means lower precision, as fp32 and fp64 are overkill for neural networks and take up lots of silicon. The current NVIDIA Pascal added fp16 and byte operations such as the DP4A convolution instruction[1]. Even smaller precision is practical (down to 1 bit with XNORnet[2], and the DoReFa paper[3] gives an excellent summary of the falloff in accuracy through 32-8-4-2-1 bits for weights, activations, and gradients). [1] https://devblogs.nvidia.com/parallelforall/mixed-precision-programming-cuda-8/ https://devblogs.nvidia.com/parallelforall/mixed-precision-p... [2] XNORnet, https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] DoReFa, https://arxiv.org/abs/1606.06160 https://arxiv.org/abs/1606.06160
- vorotato 10y ago
  Something tells me you overbought into the marketing speak
- anigbrowl 10y ago
  Seems like the sort of thing FPGAs could be good for. I have a few synthesizers that use them; elaborate feedback loops are a staple of sound design.
- deepnotderp 10y ago
  XNOR and DoReFa take pretty big hits though, and they can't be easily trained without full precision, or at least "shadow weights".
  - paulsutter 10y ago
    For XNORnet, yes, but DoReFa has training with low bitwidth activations[1]. I don't cite these papers as proof that few bits are needed, but as evidence that precision is an open question. [1] "We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers" (from the abstract)
    - p1esk 10y ago
      DoReFa needs a custom scaling factor for each image when quantizing gradients.
      - deepnotderp 10y ago
        What's wrong with that?
        p1esk 10y ago
        Their formula for quantizing gradients involves quite a bit of extra computation. It's not clear to me how much it complicates an actual implementation (in software, or hardware). To me, the most interesting question is if we can do training using just 8 bits (weights, activations, and gradients), without all that acrobatics. If so, then we can get another significant (and free!) speed up from GPUs.
  - emcq 10y ago
    What's wrong with training in full precision? There's a huge market for efficient inference in robotics, devices, and autonomous vehicles.
- deepnotderp 10y ago
  This is a common fallacy, although the architectures themselves may be changing (quite a bit!), the basic computationally intensive operations, such as convolutions and matrix multiplies aren't. The simple way is to switch from FP32 to 16-bit fixed point, and you're good to go, and you just saved almost 10X power. This is the strategy Nervana/Intel, even Nvidia, and other startups such as Wave are pursuing.
  - paulsutter 10y ago
    Exactly - evolving network architectures suggest using a non-neuromorphic design (such as Xavier, Lake Crest), which trades less precision for more cores given the same power/real estate. Neuromorphic designs like IBM's True North are more hardwired, and that's the limitation towards general purpose use. Yann LeCun's remarks on True North: https://www.facebook.com/yann.lecun/posts/10152184295832143 https://www.facebook.com/yann.lecun/posts/10152184295832143
    - deepnotderp 10y ago
      Yeah, TrueNorth is pointless.
    - deepnotderp 10y ago
      Also, just a sidenote, Xavier is based on Volta and is a while back.
    - emcq 10y ago
      The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1] LeCun's primary critique is that binary wont work; "to get good results on a task like ImageNet you need about 8 bit of precision on the neuron states". There was no evidence for his claim then, and now this is clearly false [2, 3]. LeCun's post is based more in pride than reason; he spends most of the time talking about NeuFlow which at one point was a competitor to True North for funding. In the end, NeuFlow never became a chip, but True North did. [0] https://papers.nips.cc/paper/5862-backpropagation-for-energy-efficient-neuromorphic-computing https://papers.nips.cc/paper/5862-backpropagation-for-energy... [1] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf [2] https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830
- deleted 10y ago
  [deleted]
- russdill 10y ago
  I think the Intel neuromorphic computing will be a combination of 3D XPoint and technologies gained by purchasing Altera, including routing, DSPs, and reconfigurablity.
- emcq 10y ago
  For training networks, designing en efficient chip is a scary proposition. However for inference tasks, low precision GEMM (as you said) goes a long way and better than what you often get. That's why chips like Movidius' Myriad are getting popular and are more similar to DSPs than neuromorphic designs. I agree that Intel's neuromorphic group doesn't get it, but other groups have taken neuromophic design principles that lead to efficient designs. For example, TrueNorth is very low precision, has great data locality, and though it was designed over 5 years ago can still use modern convolutional networks only imagined afterwards [0]. But its silicon implementation is not very brain like. [0] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf
m3kw9 10y ago
Seem like a fancy name for parallel computing chip that specialize in efficient parallel computing
- adevine 10y ago
  It's that what it actually is? Not familiar with the field, so is "neuromorphic chip design" really "we wired a bunch of GPUs together?" Don't mean to minimize the work involved, just trying to decipher the marketing speak.
  - DesiLurker 10y ago
    neuromorphic chips try to mimic human brain's neuronal structure for computing, entirely new paradigm. checkout IBM truNorth architecture.
  - mrstone 10y ago
    They are a chip architecture that communicates similarly to how neurons do - i.e. spiking behaviours. Makes it easy to approximate some neural systems (such as a neuromorphic retina).
  - pzone 10y ago
    I think the point is that even GPUs are suboptimal for neural network computation and we need something more specialized.
stefantalpalaru 10y ago
> A model of the brain in silicon That'd be nice, but we don't have a model of the brain right now. Artificial neural networks are so far removed from biological ones that the name is nothing more than a historical curiosity.
justinpombrio 10y ago
Could someone explain what a neuromorphic chip is? The article assumes we know, but I haven't heard of it. And the wikipedia article on "neuromorphic engineering" talks about stuff like analog circuits, copying how neurons work, and memristors, none of which seem that related.
- mattlevan 10y ago
  All of the subjects you mention at the end of your comment, especially memristors, are indeed the focus of neuromorphic computing. It's a nascent technology field, but there are plenty of papers available on IEEExplore or ACM Digital Library to satisfy curiosity! I'll take a look at the survey paper I wrote a couple summers back-- if it's decent I'll edit this post with a link. Edit, links: Paper: https://docs.google.com/file/d/0B7QHR9a8j1iiU3RxSHZSNFh2cEdvX3haUzBFWHZBNXFmYlVB/edit?usp=docslist_api&filetype=msword https://docs.google.com/file/d/0B7QHR9a8j1iiU3RxSHZSNFh2cEdv... Slides: https://docs.google.com/file/d/0B7QHR9a8j1iiSE1ET2ZTb09aNFBPZmFHOFpVRVVYalhpSG9v/edit?usp=docslist_api&filetype=mspresentation https://docs.google.com/file/d/0B7QHR9a8j1iiSE1ET2ZTb09aNFBP...
- mljoe 10y ago
  The best definition I can come up with is hardware that implements neural network architecture directly, especially McCulloch–Pitts spiking neurons (which have a temporal component). In neuromorphic chips, neurons are an actual component in the hardware, you can ask questions like "how many neurons does this chip have?". Contrast to how neural nets as we use them today which are actually implemented as a computation graph on tensors. It turns out that a special kind of neural network can be abstracted well as a series of tensor ops (dense feedforward layered networks[1]), but this is not necessarily the case for any neural network. So neuromorphic chips have a possibility of being far more general. [1]: Which are wired something like this: http://neuralnetworksanddeeplearning.com/images/tikz40.png http://neuralnetworksanddeeplearning.com/images/tikz40.png - Notice that dense connections and layered architecture. For all intents and purposes, this what neural nets look like today because of how easily it is to treat a NN with this specific wiring as a chain of tensor computations and thus execute on more conventional hardware.
- return0 10y ago
espeed 10y ago
Are these the Neuromorphic chips [1] Jeff Hawkins of Numenta [2] has been talking about? [1] Neuromorphic Chips https://www.technologyreview.com/s/526506/neuromorphic-chips/ https://www.technologyreview.com/s/526506/neuromorphic-chips... [2] Numenta papers/videos http://numenta.com/papers-videos-and-more/ http://numenta.com/papers-videos-and-more/ It would also be good to see a major chip manufacturer or cloud provider that makes its own chips (Google/IBM) get serious about graph processing chips [3,4] and moving beyond floating point [5]. [3] Novel Graph Processor Architecture https://www.ll.mit.edu/publications/journal/pdf/vol20_no1/20_1_7_Song.pdf https://www.ll.mit.edu/publications/journal/pdf/vol20_no1/20... [4] Novel Graph Processor Architecture, Prototype System, and Results https://arxiv.org/pdf/1607.06541.pdf https://arxiv.org/pdf/1607.06541.pdf [5] Stanford Seminar: Beyond Floating Point: Next Generation Computer Arithmetic https://www.youtube.com/watch?v=aP0Y1uAA-2Y https://www.youtube.com/watch?v=aP0Y1uAA-2Y
mrfusion 10y ago
Would it be possible to build an analog neural network with hardware?
- emcq 10y ago
  Yes, this is traditionally a big effort in the neuromorphic community originating in Carver Meads lab at Caltech. Perhaps the best success of such analog computation efforts are with Neurogrid (full disclosure I worked with this group): https://web.stanford.edu/group/brainsinsilicon/ https://web.stanford.edu/group/brainsinsilicon/ One thing to keep in mind is that communication is often still in digital spikes, which some may argue is more neuron like than an analog encoding.
- deepnotderp 10y ago
  Yes, I work at a startup that's doing this, but it's very dangerous with the amounts of noise, which seem fine on MNIST and even CIFAR, but you die on imagenet. The key to circumventing this is very complicated and our "secret sauce".
  - 38kkdiu 10y ago
    Can you say a bit more about what you mean by "it's very dangerous with the amounts of noise"?
    - deepnotderp 10y ago
      Network accuracy crashes
- adamnemecek 10y ago
  I'm hoping that we could see photonic computers.
return0 10y ago
Do 'Neuromorphic' chips do anything useful? At least neural networks have well known utility, but afaik, neuromorphic (i.e. heavily simplified models of neurons that one only hopes - but cannot prove are correct) have no useful applications - or even theoretical functions.
- bobsil1 10y ago
  Low-power video object recognition for military drones, security cams, etc.
aj7 10y ago
For laughs, I searched this document for the term Nvidia.

149 points by Katydid 10y ago

emcq 10y ago

The near future is not in putting learning inside a chip. We're a long ways off from the one shot learning needed to make a device actually interesting with localized learning. Instead, the future is recording and uploading your observations to the cloud, with data scientist and neural net wizards training over this dataset on a cluster with tons of GPUs, and then deploying an optimized model to scrappy low precision inference chips. This is why FPGA based designs will fail to be stunning. Specialized low precision ASICs more similar to DSPs like Movidius' Myriad (in the Phantom drones and Google's Project Tango devices), Google's TPU, upcoming Qualcomm chips, or Nervana's will become increasingly popular.

paulsutter 10y ago

The article misses the core issue: neural network architectures are still in flux. "Neuromorphic" chips are hardwired to one architecture, which makes them power efficient but less flexible. When designs are more stable, such chips could be more practical. Meanwhile, upcoming (non-neuromorphic) AI processors are taking two directions: larger numbers of simplified GPU-type cores (such as NVIDIA Xavier and Intel's Lake Crest/Nervana chips), and FPGAs. Simplifying cores means lower precision, as fp32 and fp64 are overkill for neural networks and take up lots of silicon. The current NVIDIA Pascal added fp16 and byte operations such as the DP4A convolution instruction[1]. Even smaller precision is practical (down to 1 bit with XNORnet[2], and the DoReFa paper[3] gives an excellent summary of the falloff in accuracy through 32-8-4-2-1 bits for weights, activations, and gradients). [1] https://devblogs.nvidia.com/parallelforall/mixed-precision-programming-cuda-8/ https://devblogs.nvidia.com/parallelforall/mixed-precision-p... [2] XNORnet, https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] DoReFa, https://arxiv.org/abs/1606.06160 https://arxiv.org/abs/1606.06160

vorotato 10y ago
Something tells me you overbought into the marketing speak
anigbrowl 10y ago
Seems like the sort of thing FPGAs could be good for. I have a few synthesizers that use them; elaborate feedback loops are a staple of sound design.
deepnotderp 10y ago
XNOR and DoReFa take pretty big hits though, and they can't be easily trained without full precision, or at least "shadow weights".
- paulsutter 10y ago
  For XNORnet, yes, but DoReFa has training with low bitwidth activations[1]. I don't cite these papers as proof that few bits are needed, but as evidence that precision is an open question. [1] "We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers" (from the abstract)
  - p1esk 10y ago
    DoReFa needs a custom scaling factor for each image when quantizing gradients.
    - deepnotderp 10y ago
      What's wrong with that?
      - p1esk 10y ago
        Their formula for quantizing gradients involves quite a bit of extra computation. It's not clear to me how much it complicates an actual implementation (in software, or hardware). To me, the most interesting question is if we can do training using just 8 bits (weights, activations, and gradients), without all that acrobatics. If so, then we can get another significant (and free!) speed up from GPUs.
- emcq 10y ago
  What's wrong with training in full precision? There's a huge market for efficient inference in robotics, devices, and autonomous vehicles.
deepnotderp 10y ago
This is a common fallacy, although the architectures themselves may be changing (quite a bit!), the basic computationally intensive operations, such as convolutions and matrix multiplies aren't. The simple way is to switch from FP32 to 16-bit fixed point, and you're good to go, and you just saved almost 10X power. This is the strategy Nervana/Intel, even Nvidia, and other startups such as Wave are pursuing.
- paulsutter 10y ago
  Exactly - evolving network architectures suggest using a non-neuromorphic design (such as Xavier, Lake Crest), which trades less precision for more cores given the same power/real estate. Neuromorphic designs like IBM's True North are more hardwired, and that's the limitation towards general purpose use. Yann LeCun's remarks on True North: https://www.facebook.com/yann.lecun/posts/10152184295832143 https://www.facebook.com/yann.lecun/posts/10152184295832143
  - deepnotderp 10y ago
    Yeah, TrueNorth is pointless.
  - deepnotderp 10y ago
    Also, just a sidenote, Xavier is based on Volta and is a while back.
  - emcq 10y ago
    The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1] LeCun's primary critique is that binary wont work; "to get good results on a task like ImageNet you need about 8 bit of precision on the neuron states". There was no evidence for his claim then, and now this is clearly false [2, 3]. LeCun's post is based more in pride than reason; he spends most of the time talking about NeuFlow which at one point was a competitor to True North for funding. In the end, NeuFlow never became a chip, but True North did. [0] https://papers.nips.cc/paper/5862-backpropagation-for-energy-efficient-neuromorphic-computing https://papers.nips.cc/paper/5862-backpropagation-for-energy... [1] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf [2] https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830
deleted 10y ago
[deleted]
russdill 10y ago
I think the Intel neuromorphic computing will be a combination of 3D XPoint and technologies gained by purchasing Altera, including routing, DSPs, and reconfigurablity.
emcq 10y ago
For training networks, designing en efficient chip is a scary proposition. However for inference tasks, low precision GEMM (as you said) goes a long way and better than what you often get. That's why chips like Movidius' Myriad are getting popular and are more similar to DSPs than neuromorphic designs. I agree that Intel's neuromorphic group doesn't get it, but other groups have taken neuromophic design principles that lead to efficient designs. For example, TrueNorth is very low precision, has great data locality, and though it was designed over 5 years ago can still use modern convolutional networks only imagined afterwards [0]. But its silicon implementation is not very brain like. [0] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf

m3kw9 10y ago

Seem like a fancy name for parallel computing chip that specialize in efficient parallel computing

adevine 10y ago
It's that what it actually is? Not familiar with the field, so is "neuromorphic chip design" really "we wired a bunch of GPUs together?" Don't mean to minimize the work involved, just trying to decipher the marketing speak.
- DesiLurker 10y ago
  neuromorphic chips try to mimic human brain's neuronal structure for computing, entirely new paradigm. checkout IBM truNorth architecture.
- mrstone 10y ago
  They are a chip architecture that communicates similarly to how neurons do - i.e. spiking behaviours. Makes it easy to approximate some neural systems (such as a neuromorphic retina).
- pzone 10y ago
  I think the point is that even GPUs are suboptimal for neural network computation and we need something more specialized.

stefantalpalaru 10y ago

> A model of the brain in silicon That'd be nice, but we don't have a model of the brain right now. Artificial neural networks are so far removed from biological ones that the name is nothing more than a historical curiosity.

justinpombrio 10y ago

Could someone explain what a neuromorphic chip is? The article assumes we know, but I haven't heard of it. And the wikipedia article on "neuromorphic engineering" talks about stuff like analog circuits, copying how neurons work, and memristors, none of which seem that related.

mattlevan 10y ago
All of the subjects you mention at the end of your comment, especially memristors, are indeed the focus of neuromorphic computing. It's a nascent technology field, but there are plenty of papers available on IEEExplore or ACM Digital Library to satisfy curiosity! I'll take a look at the survey paper I wrote a couple summers back-- if it's decent I'll edit this post with a link. Edit, links: Paper: https://docs.google.com/file/d/0B7QHR9a8j1iiU3RxSHZSNFh2cEdvX3haUzBFWHZBNXFmYlVB/edit?usp=docslist_api&filetype=msword https://docs.google.com/file/d/0B7QHR9a8j1iiU3RxSHZSNFh2cEdv... Slides: https://docs.google.com/file/d/0B7QHR9a8j1iiSE1ET2ZTb09aNFBPZmFHOFpVRVVYalhpSG9v/edit?usp=docslist_api&filetype=mspresentation https://docs.google.com/file/d/0B7QHR9a8j1iiSE1ET2ZTb09aNFBP...
mljoe 10y ago
The best definition I can come up with is hardware that implements neural network architecture directly, especially McCulloch–Pitts spiking neurons (which have a temporal component). In neuromorphic chips, neurons are an actual component in the hardware, you can ask questions like "how many neurons does this chip have?". Contrast to how neural nets as we use them today which are actually implemented as a computation graph on tensors. It turns out that a special kind of neural network can be abstracted well as a series of tensor ops (dense feedforward layered networks[1]), but this is not necessarily the case for any neural network. So neuromorphic chips have a possibility of being far more general. [1]: Which are wired something like this: http://neuralnetworksanddeeplearning.com/images/tikz40.png http://neuralnetworksanddeeplearning.com/images/tikz40.png - Notice that dense connections and layered architecture. For all intents and purposes, this what neural nets look like today because of how easily it is to treat a NN with this specific wiring as a chain of tensor computations and thus execute on more conventional hardware.
return0 10y ago

espeed 10y ago

Are these the Neuromorphic chips [1] Jeff Hawkins of Numenta [2] has been talking about? [1] Neuromorphic Chips https://www.technologyreview.com/s/526506/neuromorphic-chips/ https://www.technologyreview.com/s/526506/neuromorphic-chips... [2] Numenta papers/videos http://numenta.com/papers-videos-and-more/ http://numenta.com/papers-videos-and-more/ It would also be good to see a major chip manufacturer or cloud provider that makes its own chips (Google/IBM) get serious about graph processing chips [3,4] and moving beyond floating point [5]. [3] Novel Graph Processor Architecture https://www.ll.mit.edu/publications/journal/pdf/vol20_no1/20_1_7_Song.pdf https://www.ll.mit.edu/publications/journal/pdf/vol20_no1/20... [4] Novel Graph Processor Architecture, Prototype System, and Results https://arxiv.org/pdf/1607.06541.pdf https://arxiv.org/pdf/1607.06541.pdf [5] Stanford Seminar: Beyond Floating Point: Next Generation Computer Arithmetic https://www.youtube.com/watch?v=aP0Y1uAA-2Y https://www.youtube.com/watch?v=aP0Y1uAA-2Y

mrfusion 10y ago

Would it be possible to build an analog neural network with hardware?

emcq 10y ago
Yes, this is traditionally a big effort in the neuromorphic community originating in Carver Meads lab at Caltech. Perhaps the best success of such analog computation efforts are with Neurogrid (full disclosure I worked with this group): https://web.stanford.edu/group/brainsinsilicon/ https://web.stanford.edu/group/brainsinsilicon/ One thing to keep in mind is that communication is often still in digital spikes, which some may argue is more neuron like than an analog encoding.
deepnotderp 10y ago
Yes, I work at a startup that's doing this, but it's very dangerous with the amounts of noise, which seem fine on MNIST and even CIFAR, but you die on imagenet. The key to circumventing this is very complicated and our "secret sauce".
- 38kkdiu 10y ago
  Can you say a bit more about what you mean by "it's very dangerous with the amounts of noise"?
  - deepnotderp 10y ago
    Network accuracy crashes
adamnemecek 10y ago
I'm hoping that we could see photonic computers.

return0 10y ago

Do 'Neuromorphic' chips do anything useful? At least neural networks have well known utility, but afaik, neuromorphic (i.e. heavily simplified models of neurons that one only hopes - but cannot prove are correct) have no useful applications - or even theoretical functions.

bobsil1 10y ago
Low-power video object recognition for military drones, security cams, etc.

aj7 10y ago

For laughs, I searched this document for the term Nvidia.