4 ms·
What's the key advantage of this Nervana architecture over GPUs? I can theorycraft... hypothetically, GPUs have a memory architecture structured like this: (us
by dragontamer 9y ago
What's the key advantage of this Nervana architecture over GPUs?
I can theorycraft... hypothetically, GPUs have a memory architecture structured like this: (using OpenCL terms here) Global Memory <-> Local <-> Private Memory, correlating to the GPU <-> Work Group <-> Work Item.
On AMD's systems, "Local" and/or Work Group tier is a group of roughly 256-work items (or in CUDA Terms, a 256-"Block" of "Threads") and all 256-work items can access Local Memory at outstanding speeds (and then there's even faster "Private" memory per work item / thread, which is basically a hardware register).
On say a Vega 64, there are 64-compute units (each of which has 256 work items running in parallel). The typical way Compute Unit 1 can talk to Compute Unit 2 is to write data from CU1 into Global Memory (which is off-chip), and then read it back in in CU2. In effect, GPUs are designed for high-bandwidth communications WITHIN a Workgroup (or "Warp" in CUDA terms), but they have slow communications ACROSS Work-groups / Warps.
In effect, there's only "one" Global Memory on a GPU. And in the case of a Vega64 GPU, that's 16384 work items that might be trying to hit Global Memory at the same time. True, there's caching layers and other optimizations, but any methodology based on Global resources will naturally slow down code and hamper parallelism.
Neural Networks possibly can have faster memory message passing if there were a different architecture. Imagine if the compute-units allowed quick communication in a torus for example. So compute unit #1 can quickly communicate to compute unit #2.
This would roughly correlate to "Layer1 Neurons" passing signals to "Layer2 Neurons" and vice versa (say for backpropagation of errors).
Alas, I don't see much information on what Nervana is doing differently. When "Parallela" came out a few years ago, they were crystal clear on how their memory architecture was grossly different than a GPU... it'd be nice if Nervana's marketing material was similarly clear.
----------
Hmm, this page is a bit more technical: https://www.intelnervana.com/intel-nervana-neural-network-processors-nnp-redefine-ai-silicon/ https://www.intelnervana.com/intel-nervana-neural-network-pr...
It seems like the big selling points are:
* "Flexpoint" -- They're a bit light on the details, but they argue that "Flexpoint" is better than Floating Point. It'd be nice if they were a bit more transparent on what "Flexpoint" is, but I'll imagine that its like a Logarithmic Number System (https://en.wikipedia.org/wiki/Logarithmic_number_system https://en.wikipedia.org/wiki/Logarithmic_number_system) or similar, which probably would be better for low-precision Neural Network computations.
* "Better Memory Architecture" -- I can't find any details on why their memory architecture is better. They just sorta... claim its better.
Ultimately, GPUs were designed for graphics problems. So I'm sure there's a better architecture out there for Neural Network problems. Its just fundamentally a different kind of parallelism. (Image processing / shaders handling the top-left corner of a polygon don't need to know what's going on on the bottom-right polygon. So GPUs don't have high-bandwidth communication lines between those units. Neural Networks require a little bit more communication than image processing problems did from the past). But I'm not really seeing "why" this architecture is better yet.
- tachyonbeam 9y agoThe main competitor for this is other dedicated neural network hardware, such as the Google TPU: https://cloud.google.com/blog/big-data/2017/05/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu https://cloud.google.com/blog/big-data/2017/05/an-in-depth-l... One potential advantage is that neural network weights could be as small as 8 bits. If you're working with 8-bit quantities, you're saving in terms of memory bandwidth, power, and you can fit more computational units. Neural network hardware also isn't going to need hardware implementations of things like sine/cosine, square root. Can potentially avoid control flow and branching hardware (or have less of it). May not need the whole complex register file and thread swapping machinery that GPUs have.
- dragontamer 9y agoFlexpoint looks like its 16-bits however: https://www.intelnervana.com/nervana-engine-delivers-deep-learning-at-ludicrous-speed/ https://www.intelnervana.com/nervana-engine-delivers-deep-le... And I'm not really sure that stripping "Sin" and "Cos" out of the FPU units would really save that much power and/or improve efficiency. Generally speaking: the biggest problem today is the memory architecture of your computing device (be it a CPU, GPU, Paralella, or FPGA). Efficient CPU programming requires deep understanding of L1 / L2 / L3 / Off-chip cache. And similarly, efficient GPU code require deep understanding of the memory hierarchy. ------------------- https://www.intelnervana.com/nervana-engine-delivers-deep-learning-at-ludicrous-speed/ https://www.intelnervana.com/nervana-engine-delivers-deep-le... Looking at the Nervana posts... it looks like the "memory" question is: 1. HBM: Except both AMD Fury and AMD Vega64 are already using HBM. NVidia also has an HBM offering at the very very high end. 2. 16-bit Flexpoint: Except NVidia and AMD cards already can efficiently calculate half-floats (16-bit floats). 3. MAC: Multiply and Accumulate. AMD and NVidia cards have been optimizing this instruction and can perform Matrix Multiplications at ludicrous speeds already. Maybe the Google TPU can be a theoretical competitor... but until Google's TPU becomes available from newegg.com, it really isn't a consideration for most people. ---------------- The main "advantage" of Nervana looks like its improved memory architecture with "separate pipelines for computation and data management". (aka: vague marketing talk). And I really do think this is the key: depending on how memory is architected on the Nervana system, it really can make-or-break the speed of applications. But is also the part that's the most mysterious with the least amount of documentation so far...