3 ms·

This is a common fallacy, although the architectures themselves may be changing (quite a bit!), the basic computationally intensive operations, such as convolut

This is a common fallacy, although the architectures themselves may be changing (quite a bit!), the basic computationally intensive operations, such as convolutions and matrix multiplies aren't. The simple way is to switch from FP32 to 16-bit fixed point, and you're good to go, and you just saved almost 10X power. This is the strategy Nervana/Intel, even Nvidia, and other startups such as Wave are pursuing.

paulsutter 10y ago
Exactly - evolving network architectures suggest using a non-neuromorphic design (such as Xavier, Lake Crest), which trades less precision for more cores given the same power/real estate. Neuromorphic designs like IBM's True North are more hardwired, and that's the limitation towards general purpose use. Yann LeCun's remarks on True North: https://www.facebook.com/yann.lecun/posts/10152184295832143 https://www.facebook.com/yann.lecun/posts/10152184295832143
- deepnotderp 10y ago
  Yeah, TrueNorth is pointless.
- deepnotderp 10y ago
  Also, just a sidenote, Xavier is based on Volta and is a while back.
- emcq 10y ago
  The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1] LeCun's primary critique is that binary wont work; "to get good results on a task like ImageNet you need about 8 bit of precision on the neuron states". There was no evidence for his claim then, and now this is clearly false [2, 3]. LeCun's post is based more in pride than reason; he spends most of the time talking about NeuFlow which at one point was a competitor to True North for funding. In the end, NeuFlow never became a chip, but True North did. [0] https://papers.nips.cc/paper/5862-backpropagation-for-energy-efficient-neuromorphic-computing https://papers.nips.cc/paper/5862-backpropagation-for-energy... [1] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf [2] https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830
  - deepnotderp 10y ago
    Yeah, this all sounds nice in practice, but look at the actual numbers: https://arxiv.org/abs/1603.08270 https://arxiv.org/abs/1603.08270 1) Nothing on imagenet 2) They already fall to 83% accuracy on CIFAR-10! Imagine how bad imagenet will be! If they string many chips together (exploding their power consumption, since here comes the Von Neumann Bottleneck of data movement), they get a paltry 89%..... Meanwhile even squeezenet achieves better results.
    - emcq 10y ago
      I'm sure today we could create an _even better_ chip focused on the advancements in neural networks and chip design, but it's awesome that a chip from 2012 can still take us so far! AlexNet was just a baby then. I doubt any CPU, GPU, FPGA, or DSP from 2012 would hold up as well. I dont think you understand their architecture, and neither did LeCun. The Von Neumann Bottleneck is a specific term referring to limited throughput between data in memory and compute in the CPU. TrueNorth is not a Von Neumann architecture and does not have this bottleneck. Memory is located adjacent to compute elements in True North. For comparison, GPU's have very tiny amounts of on chip memory, and have to spend lots of energy copying data back and forth to off chip memory, which is why they are investing heavily in approaches like HBM. FPGAs also dont have as much memory as an ASIC because they need to dedicate space to reprogrammable logic, integrated ARM cores/DSPs, etc. The chips can be laid out in flexible topologies such as a grid. While it's true that communication between chips is more power intensive than within a chip, this cost is only occurred for the relatively small amount of traffic sent between chips versus computed locally. Hierarchy and small world nature of neural networks can mean that there is more local computation than you would expect naively, and a grid can mean a spike routed from one core to the furthest core would be O(sqrt(N)) instead of O(N).
      - deepnotderp 10y ago
        ... Pretty sure LeCun and I understand what the Von Neumann Bottleneck is thank you very much. The thing is though that TrueNorth isn't doing anything special by pouring a ton of memory on die, and even in GPUs, CNN runtimes and energy consumption is dominated by compute.
pizza 10y ago
see: discussion for "Why are Eight Bits Enough for Deep Neural Networks?" 511 days ago https://news.ycombinator.com/item?id=10244398 https://news.ycombinator.com/item?id=10244398

paulsutter 10y ago

Exactly - evolving network architectures suggest using a non-neuromorphic design (such as Xavier, Lake Crest), which trades less precision for more cores given the same power/real estate. Neuromorphic designs like IBM's True North are more hardwired, and that's the limitation towards general purpose use. Yann LeCun's remarks on True North: https://www.facebook.com/yann.lecun/posts/10152184295832143 https://www.facebook.com/yann.lecun/posts/10152184295832143

deepnotderp 10y ago
Yeah, TrueNorth is pointless.
deepnotderp 10y ago
Also, just a sidenote, Xavier is based on Volta and is a while back.
emcq 10y ago
The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1] LeCun's primary critique is that binary wont work; "to get good results on a task like ImageNet you need about 8 bit of precision on the neuron states". There was no evidence for his claim then, and now this is clearly false [2, 3]. LeCun's post is based more in pride than reason; he spends most of the time talking about NeuFlow which at one point was a competitor to True North for funding. In the end, NeuFlow never became a chip, but True North did. [0] https://papers.nips.cc/paper/5862-backpropagation-for-energy-efficient-neuromorphic-computing https://papers.nips.cc/paper/5862-backpropagation-for-energy... [1] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf [2] https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830
- deepnotderp 10y ago
  Yeah, this all sounds nice in practice, but look at the actual numbers: https://arxiv.org/abs/1603.08270 https://arxiv.org/abs/1603.08270 1) Nothing on imagenet 2) They already fall to 83% accuracy on CIFAR-10! Imagine how bad imagenet will be! If they string many chips together (exploding their power consumption, since here comes the Von Neumann Bottleneck of data movement), they get a paltry 89%..... Meanwhile even squeezenet achieves better results.
  - emcq 10y ago
    I'm sure today we could create an _even better_ chip focused on the advancements in neural networks and chip design, but it's awesome that a chip from 2012 can still take us so far! AlexNet was just a baby then. I doubt any CPU, GPU, FPGA, or DSP from 2012 would hold up as well. I dont think you understand their architecture, and neither did LeCun. The Von Neumann Bottleneck is a specific term referring to limited throughput between data in memory and compute in the CPU. TrueNorth is not a Von Neumann architecture and does not have this bottleneck. Memory is located adjacent to compute elements in True North. For comparison, GPU's have very tiny amounts of on chip memory, and have to spend lots of energy copying data back and forth to off chip memory, which is why they are investing heavily in approaches like HBM. FPGAs also dont have as much memory as an ASIC because they need to dedicate space to reprogrammable logic, integrated ARM cores/DSPs, etc. The chips can be laid out in flexible topologies such as a grid. While it's true that communication between chips is more power intensive than within a chip, this cost is only occurred for the relatively small amount of traffic sent between chips versus computed locally. Hierarchy and small world nature of neural networks can mean that there is more local computation than you would expect naively, and a grid can mean a spike routed from one core to the furthest core would be O(sqrt(N)) instead of O(N).
    - deepnotderp 10y ago
      ... Pretty sure LeCun and I understand what the Von Neumann Bottleneck is thank you very much. The thing is though that TrueNorth isn't doing anything special by pouring a ton of memory on die, and even in GPUs, CNN runtimes and energy consumption is dominated by compute.

pizza 10y ago

see: discussion for "Why are Eight Bits Enough for Deep Neural Networks?" 511 days ago https://news.ycombinator.com/item?id=10244398 https://news.ycombinator.com/item?id=10244398