4 ms·

The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1]

by emcq 10y ago

The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1] LeCun's primary critique is that binary wont work; "to get good results on a task like ImageNet you need about 8 bit of precision on the neuron states". There was no evidence for his claim then, and now this is clearly false [2, 3]. LeCun's post is based more in pride than reason; he spends most of the time talking about NeuFlow which at one point was a competitor to True North for funding. In the end, NeuFlow never became a chip, but True North did. [0] https://papers.nips.cc/paper/5862-backpropagation-for-energy-efficient-neuromorphic-computing https://papers.nips.cc/paper/5862-backpropagation-for-energy... [1] https://arxiv.org/pdf/1603.08270.pdf https://arxiv.org/pdf/1603.08270.pdf [2] https://arxiv.org/abs/1603.05279 https://arxiv.org/abs/1603.05279 [3] https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830

deepnotderp 10y ago
Yeah, this all sounds nice in practice, but look at the actual numbers: https://arxiv.org/abs/1603.08270 https://arxiv.org/abs/1603.08270 1) Nothing on imagenet 2) They already fall to 83% accuracy on CIFAR-10! Imagine how bad imagenet will be! If they string many chips together (exploding their power consumption, since here comes the Von Neumann Bottleneck of data movement), they get a paltry 89%..... Meanwhile even squeezenet achieves better results.
- emcq 10y ago
  I'm sure today we could create an _even better_ chip focused on the advancements in neural networks and chip design, but it's awesome that a chip from 2012 can still take us so far! AlexNet was just a baby then. I doubt any CPU, GPU, FPGA, or DSP from 2012 would hold up as well. I dont think you understand their architecture, and neither did LeCun. The Von Neumann Bottleneck is a specific term referring to limited throughput between data in memory and compute in the CPU. TrueNorth is not a Von Neumann architecture and does not have this bottleneck. Memory is located adjacent to compute elements in True North. For comparison, GPU's have very tiny amounts of on chip memory, and have to spend lots of energy copying data back and forth to off chip memory, which is why they are investing heavily in approaches like HBM. FPGAs also dont have as much memory as an ASIC because they need to dedicate space to reprogrammable logic, integrated ARM cores/DSPs, etc. The chips can be laid out in flexible topologies such as a grid. While it's true that communication between chips is more power intensive than within a chip, this cost is only occurred for the relatively small amount of traffic sent between chips versus computed locally. Hierarchy and small world nature of neural networks can mean that there is more local computation than you would expect naively, and a grid can mean a spike routed from one core to the furthest core would be O(sqrt(N)) instead of O(N).
  - deepnotderp 10y ago
    ... Pretty sure LeCun and I understand what the Von Neumann Bottleneck is thank you very much. The thing is though that TrueNorth isn't doing anything special by pouring a ton of memory on die, and even in GPUs, CNN runtimes and energy consumption is dominated by compute.
    - emcq 10y ago
      I appreciate healthy discussion of technical topics. However, I'm not sure you're having this discussion in good faith. I wrote this response in case you are. LeCun never said anything about the Von Neumann Bottleneck. TrueNorth is not a Von Neumann architecture; it does not have a memory bus; it does not have the Von Neumann Bottleneck [0,2,3]. From wikipedia [1]: "TrueNorth circumvents the von-Neumann-architecture bottlenecks and is very energy-efficient, consuming 70 milliwatts, about 1/10,000th the power density of conventional microprocessors" If you disagree, please explain how you think the Von Neumann Bottleneck applies here. With regards to energy consumption keep in mind the smallest GPUs (TX1) are ~10W, typical FGPAs ~1W, versus 70mW for TrueNorth! It's popular to hate on TrueNorth but you could throw 10 of them together and still be fantastically more efficient than anything else today - that's super cool to me! It required lots of special engineering effort to get right, such as building a lot of on chip memory. On chip memory is one of the most difficult components to get right, minimizing transistors while not breaking physics. It's not as simple as "pouring tons of memory on a die" and requires specialized engineers that hand-layout these components. The event driven asynchronous nature of TrueNorth is fairly unique and undoubtedly added complexity to the memory design. Do you have any references or evidence for CNN runtimes being mostly dominated by compute? The operations performed in a CNN are more than just convolution; for every input you multiply by a weight, you now have a memory bound problem, which is much more expensive than ALU operations. Don't just take my word for it, listen to Bill Dally (Chief Scientist at NVIDIA, Stanford CS prof, and general computer architecture badass) [4]: "State-of-the-art deep neural networks (DNNs) have hundreds of millions of connections and are both computationally and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources and power budgets. While custom hardware helps the computation, fetching weights from DRAM is two orders of magnitude more expensive than ALU operations, and dominates the required power." This is what TrueNorth got right, and made its bet completing design before AlexNet was published. That was a time where Hinton was viewed by the ML community as a heretic talking about RBMs and backprop and hardly anyone believed him. TrueNorth, like NNs at the time, gets some shade by doing things differently that over time we're seeing validated by other researchers and architectures incorporating. I recommend reading [4] if you haven't already, as it is rich in insights for building efficient NN architectures. [0] https://en.wikipedia.org/wiki/Von_Neumann_architecture#Von_Neumann_bottleneck https://en.wikipedia.org/wiki/Von_Neumann_architecture#Von_N... [1] https://en.wikipedia.org/wiki/TrueNorth https://en.wikipedia.org/wiki/TrueNorth [2] http://ieeexplore.ieee.org/document/7229264/?reload=true&arnumber=7229264 http://ieeexplore.ieee.org/document/7229264/?reload=true&arn... [3] http://www.research.ibm.com/articles/brain-chip.shtml http://www.research.ibm.com/articles/brain-chip.shtml [4] https://arxiv.org/pdf/1602.01528.pdf https://arxiv.org/pdf/1602.01528.pdf

by emcq 10y ago

deepnotderp 10y ago

Yeah, this all sounds nice in practice, but look at the actual numbers: https://arxiv.org/abs/1603.08270 https://arxiv.org/abs/1603.08270 1) Nothing on imagenet 2) They already fall to 83% accuracy on CIFAR-10! Imagine how bad imagenet will be! If they string many chips together (exploding their power consumption, since here comes the Von Neumann Bottleneck of data movement), they get a paltry 89%..... Meanwhile even squeezenet achieves better results.

emcq 10y ago
I'm sure today we could create an _even better_ chip focused on the advancements in neural networks and chip design, but it's awesome that a chip from 2012 can still take us so far! AlexNet was just a baby then. I doubt any CPU, GPU, FPGA, or DSP from 2012 would hold up as well. I dont think you understand their architecture, and neither did LeCun. The Von Neumann Bottleneck is a specific term referring to limited throughput between data in memory and compute in the CPU. TrueNorth is not a Von Neumann architecture and does not have this bottleneck. Memory is located adjacent to compute elements in True North. For comparison, GPU's have very tiny amounts of on chip memory, and have to spend lots of energy copying data back and forth to off chip memory, which is why they are investing heavily in approaches like HBM. FPGAs also dont have as much memory as an ASIC because they need to dedicate space to reprogrammable logic, integrated ARM cores/DSPs, etc. The chips can be laid out in flexible topologies such as a grid. While it's true that communication between chips is more power intensive than within a chip, this cost is only occurred for the relatively small amount of traffic sent between chips versus computed locally. Hierarchy and small world nature of neural networks can mean that there is more local computation than you would expect naively, and a grid can mean a spike routed from one core to the furthest core would be O(sqrt(N)) instead of O(N).
- deepnotderp 10y ago
  ... Pretty sure LeCun and I understand what the Von Neumann Bottleneck is thank you very much. The thing is though that TrueNorth isn't doing anything special by pouring a ton of memory on die, and even in GPUs, CNN runtimes and energy consumption is dominated by compute.
  - emcq 10y ago
    I appreciate healthy discussion of technical topics. However, I'm not sure you're having this discussion in good faith. I wrote this response in case you are. LeCun never said anything about the Von Neumann Bottleneck. TrueNorth is not a Von Neumann architecture; it does not have a memory bus; it does not have the Von Neumann Bottleneck [0,2,3]. From wikipedia [1]: "TrueNorth circumvents the von-Neumann-architecture bottlenecks and is very energy-efficient, consuming 70 milliwatts, about 1/10,000th the power density of conventional microprocessors" If you disagree, please explain how you think the Von Neumann Bottleneck applies here. With regards to energy consumption keep in mind the smallest GPUs (TX1) are ~10W, typical FGPAs ~1W, versus 70mW for TrueNorth! It's popular to hate on TrueNorth but you could throw 10 of them together and still be fantastically more efficient than anything else today - that's super cool to me! It required lots of special engineering effort to get right, such as building a lot of on chip memory. On chip memory is one of the most difficult components to get right, minimizing transistors while not breaking physics. It's not as simple as "pouring tons of memory on a die" and requires specialized engineers that hand-layout these components. The event driven asynchronous nature of TrueNorth is fairly unique and undoubtedly added complexity to the memory design. Do you have any references or evidence for CNN runtimes being mostly dominated by compute? The operations performed in a CNN are more than just convolution; for every input you multiply by a weight, you now have a memory bound problem, which is much more expensive than ALU operations. Don't just take my word for it, listen to Bill Dally (Chief Scientist at NVIDIA, Stanford CS prof, and general computer architecture badass) [4]: "State-of-the-art deep neural networks (DNNs) have hundreds of millions of connections and are both computationally and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources and power budgets. While custom hardware helps the computation, fetching weights from DRAM is two orders of magnitude more expensive than ALU operations, and dominates the required power." This is what TrueNorth got right, and made its bet completing design before AlexNet was published. That was a time where Hinton was viewed by the ML community as a heretic talking about RBMs and backprop and hardly anyone believed him. TrueNorth, like NNs at the time, gets some shade by doing things differently that over time we're seeing validated by other researchers and architectures incorporating. I recommend reading [4] if you haven't already, as it is rich in insights for building efficient NN architectures. [0] https://en.wikipedia.org/wiki/Von_Neumann_architecture#Von_Neumann_bottleneck https://en.wikipedia.org/wiki/Von_Neumann_architecture#Von_N... [1] https://en.wikipedia.org/wiki/TrueNorth https://en.wikipedia.org/wiki/TrueNorth [2] http://ieeexplore.ieee.org/document/7229264/?reload=true&arnumber=7229264 http://ieeexplore.ieee.org/document/7229264/?reload=true&arn... [3] http://www.research.ibm.com/articles/brain-chip.shtml http://www.research.ibm.com/articles/brain-chip.shtml [4] https://arxiv.org/pdf/1602.01528.pdf https://arxiv.org/pdf/1602.01528.pdf