5 ms·

The article misses the core issue: neural network architectures are still in flux. "Neuromorphic" chips are hardwired to one architecture, which makes them powe

The article misses the core issue: neural network architectures are still in flux. "Neuromorphic" chips are hardwired to one architecture, which makes them power efficient but less flexible. When designs are more stable, such chips could be more practical. Meanwhile, upcoming (non-neuromorphic) AI processors are taking two directions: larger numbers of simplified GPU-type cores (such as NVIDIA Xavier and Intel's Lake Crest/Nervana chips), and FPGAs. Simplifying cores means lower precision, as fp32 and fp64 are overkill for neural networks and take up lots of silicon. The current NVIDIA Pascal added fp16 and byte operations such as the DP4A convolution instruction[1]. Even smaller precision is practical (down to 1 bit with XNORnet[2], and the DoReFa paper[3] gives an excellent summary of the falloff in accuracy through 32-8-4-2-1 bits for weights, activations, and gradients). [1] https://devblogs.nvidia.com/parallelforall/mixed-precision-programming-cuda-8/ https://devblogs.nvidia.com/parallelforall/mixed-precision-p... [2] XNORnet, https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] DoReFa, https://arxiv.org/abs/1606.06160 https://arxiv.org/abs/1606.06160

vorotato 10y ago
Something tells me you overbought into the marketing speak
anigbrowl 10y ago
Seems like the sort of thing FPGAs could be good for. I have a few synthesizers that use them; elaborate feedback loops are a staple of sound design.
deepnotderp 10y ago
XNOR and DoReFa take pretty big hits though, and they can't be easily trained without full precision, or at least "shadow weights".
- paulsutter 10y ago
  For XNORnet, yes, but DoReFa has training with low bitwidth activations[1]. I don't cite these papers as proof that few bits are needed, but as evidence that precision is an open question. [1] "We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers" (from the abstract)
  - p1esk 10y ago
    DoReFa needs a custom scaling factor for each image when quantizing gradients.
    - deepnotderp 10y ago
      What's wrong with that?
      - p1esk 10y ago
        Their formula for quantizing gradients involves quite a bit of extra computation. It's not clear to me how much it complicates an actual implementation (in software, or hardware). To me, the most interesting question is if we can do training using just 8 bits (weights, activations, and gradients), without all that acrobatics. If so, then we can get another significant (and free!) speed up from GPUs.
- emcq 10y ago
  What's wrong with training in full precision? There's a huge market for efficient inference in robotics, devices, and autonomous vehicles.
deepnotderp 10y ago
This is a common fallacy, although the architectures themselves may be changing (quite a bit!), the basic computationally intensive operations, such as convolutions and matrix multiplies aren't. The simple way is to switch from FP32 to 16-bit fixed point, and you're good to go, and you just saved almost 10X power. This is the strategy Nervana/Intel, even Nvidia, and other startups such as Wave are pursuing.
- paulsutter 10y ago
  Exactly - evolving network architectures suggest using a non-neuromorphic design (such as Xavier, Lake Crest), which trades less precision for more cores given the same power/real estate. Neuromorphic designs like IBM's True North are more hardwired, and that's the limitation towards general purpose use. Yann LeCun's remarks on True North: https://www.facebook.com/yann.lecun/posts/10152184295832143 https://www.facebook.com/yann.lecun/posts/10152184295832143
  - deepnotderp 10y ago
    Yeah, TrueNorth is pointless.
  - deepnotderp 10y ago
    Also, just a sidenote, Xavier is based on Volta and is a while back.
  - emcq 10y ago
    The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1] LeCun's primary critique is that binary wont work; "to get good results on a task like ImageNet you need about 8 bit of precision on the neuron states". There was no evidence for his claim then, and now this is clearly false [2, 3]. LeCun's post is based more in pride than reason; he spends most of the time talking about NeuFlow which at one point was a competitor to True North for funding. In the end, NeuFlow never became a chip, but True North did. [0] https://papers.nips.cc/paper/5862-backpropagation-for-energy-efficient-neuromorphic-computing https://papers.nips.cc/paper/5862-backpropagation-for-energy... [1] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf [2] https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830
deleted 10y ago
[deleted]
russdill 10y ago
I think the Intel neuromorphic computing will be a combination of 3D XPoint and technologies gained by purchasing Altera, including routing, DSPs, and reconfigurablity.
emcq 10y ago
For training networks, designing en efficient chip is a scary proposition. However for inference tasks, low precision GEMM (as you said) goes a long way and better than what you often get. That's why chips like Movidius' Myriad are getting popular and are more similar to DSPs than neuromorphic designs. I agree that Intel's neuromorphic group doesn't get it, but other groups have taken neuromophic design principles that lead to efficient designs. For example, TrueNorth is very low precision, has great data locality, and though it was designed over 5 years ago can still use modern convolutional networks only imagined afterwards [0]. But its silicon implementation is not very brain like. [0] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf

vorotato 10y ago

Something tells me you overbought into the marketing speak

anigbrowl 10y ago

Seems like the sort of thing FPGAs could be good for. I have a few synthesizers that use them; elaborate feedback loops are a staple of sound design.

deepnotderp 10y ago

XNOR and DoReFa take pretty big hits though, and they can't be easily trained without full precision, or at least "shadow weights".

paulsutter 10y ago
For XNORnet, yes, but DoReFa has training with low bitwidth activations[1]. I don't cite these papers as proof that few bits are needed, but as evidence that precision is an open question. [1] "We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers" (from the abstract)
- p1esk 10y ago
  DoReFa needs a custom scaling factor for each image when quantizing gradients.
  - deepnotderp 10y ago
    What's wrong with that?
    - p1esk 10y ago
      Their formula for quantizing gradients involves quite a bit of extra computation. It's not clear to me how much it complicates an actual implementation (in software, or hardware). To me, the most interesting question is if we can do training using just 8 bits (weights, activations, and gradients), without all that acrobatics. If so, then we can get another significant (and free!) speed up from GPUs.
emcq 10y ago
What's wrong with training in full precision? There's a huge market for efficient inference in robotics, devices, and autonomous vehicles.

deepnotderp 10y ago

This is a common fallacy, although the architectures themselves may be changing (quite a bit!), the basic computationally intensive operations, such as convolutions and matrix multiplies aren't. The simple way is to switch from FP32 to 16-bit fixed point, and you're good to go, and you just saved almost 10X power. This is the strategy Nervana/Intel, even Nvidia, and other startups such as Wave are pursuing.

paulsutter 10y ago
Exactly - evolving network architectures suggest using a non-neuromorphic design (such as Xavier, Lake Crest), which trades less precision for more cores given the same power/real estate. Neuromorphic designs like IBM's True North are more hardwired, and that's the limitation towards general purpose use. Yann LeCun's remarks on True North: https://www.facebook.com/yann.lecun/posts/10152184295832143 https://www.facebook.com/yann.lecun/posts/10152184295832143
- deepnotderp 10y ago
  Yeah, TrueNorth is pointless.
- deepnotderp 10y ago
  Also, just a sidenote, Xavier is based on Volta and is a while back.
- emcq 10y ago
  The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1] LeCun's primary critique is that binary wont work; "to get good results on a task like ImageNet you need about 8 bit of precision on the neuron states". There was no evidence for his claim then, and now this is clearly false [2, 3]. LeCun's post is based more in pride than reason; he spends most of the time talking about NeuFlow which at one point was a competitor to True North for funding. In the end, NeuFlow never became a chip, but True North did. [0] https://papers.nips.cc/paper/5862-backpropagation-for-energy-efficient-neuromorphic-computing https://papers.nips.cc/paper/5862-backpropagation-for-energy... [1] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf [2] https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830

deleted 10y ago

[deleted]

russdill 10y ago

I think the Intel neuromorphic computing will be a combination of 3D XPoint and technologies gained by purchasing Altera, including routing, DSPs, and reconfigurablity.

emcq 10y ago

For training networks, designing en efficient chip is a scary proposition. However for inference tasks, low precision GEMM (as you said) goes a long way and better than what you often get. That's why chips like Movidius' Myriad are getting popular and are more similar to DSPs than neuromorphic designs. I agree that Intel's neuromorphic group doesn't get it, but other groups have taken neuromophic design principles that lead to efficient designs. For example, TrueNorth is very low precision, has great data locality, and though it was designed over 5 years ago can still use modern convolutional networks only imagined afterwards [0]. But its silicon implementation is not very brain like. [0] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf