6 ms·
Hardware Architectures for Deep Neural Networks [pdf]
- nshm 9y agoThere is also a paper from same authors with about the same content: Efficient Processing of Deep Neural Networks: A Tutorial and Survey Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, Joel Emer https://arxiv.org/abs/1703.09039 https://arxiv.org/abs/1703.09039
- Shishram 9y agolink seems broken!
- daubi 9y agoNot broken for me - it just needs a while to load, the file is huge
- hacker_9 9y agoIt's a 32MB PDF, as it's image heavy, and takes a while to download from their servers.
- binaryzeitgeist 9y ago290 image-heavy pages. smh
- neolefty 9y agoI've been wanting to know how electronics' power efficiency compares to biological neurons, and this paper gives a clue. The most efficient hardware it mentions is the GV100 "Tensor Core", at 400GFLOPS/W for FP16. If a typical neuron requires 10^6 ATP per activation [1], and if it takes 30.5 kJ/mol to charge ATP [2], and if a typical neuron has 100 axons, each of which is contributing one FLOP, then I _think_ a human neuron is about 500 times as efficient as a GV100 [3], at 200,000 GFLOPS per watt. [1] https://www.extremetech.com/extreme/185984-the-human-brains-remarkably-low-power-consumption-and-how-computers-might-mimic-its-efficiency https://www.extremetech.com/extreme/185984-the-human-brains-... [2] https://en.wikipedia.org/wiki/Adenosine_triphosphate https://en.wikipedia.org/wiki/Adenosine_triphosphate [3] Neuron: 10E6 ATP = 1 activation = 100 FLOP 30.5 kJ = 1 mole ATP = 6E23 ATP 1 kJ = 0.28 Wh ... 2E5 GFLOPS = 1 W GV100 "Tensor Core": 4E2 GFLOPS = 1 W
- bga 9y agoKeep in mind the energy costs of the supporting hardware for that brain, though.
- TeMPOraL 9y agoComputers have supporting hardware too - powerplants (plus related infrastructure) and cooling systems, at minimum.
- mannigfaltig 9y agoHumans consume 0.7-1.3 kW per capita. A network of these chips scaled to human brain capacity would consume around 10 kW. It is really not a big difference. Our robot overlords are going to be quite efficient!
- deepnotderp 9y agoWell, keep in mind that you should never really use the advertised number as your comparison point, because that's more the "highest that you'll ever see" rather than the "average reliable performance". Not to mention they assume everything magically is in the caches as far as memory is concerned, which is absolutely untrue.
- neolefty 9y agoYes, there are many confounding factors and sources of uncertainty on both sides -- here are a few I can think of: * memory isn't accounted for * FLOP per activation is estimated at 100 * neurons can vary by a factor of ~1000 in how much energy they require to activate, based on their size and speed * ATP: I'm not really sure what to make of the energy here; I used 30kJ / mol. Wikipedia says "The hydrolysis of ATP into ADP and inorganic phosphate releases 30.5 kJ/mol of enthalpy, with a change in free energy of 3.4 kJ/mol."[1] What's the efficiency of our digestive & recharging systems -- maybe 5% - 30%? https://en.wikipedia.org/wiki/Adenosine_triphosphate https://en.wikipedia.org/wiki/Adenosine_triphosphate
- deleted 9y ago