4 ms·
Being computationally limited by GEMM is both a benefit and a pitfall. The operation itself is highly parallelizable and typically compute-bound (as opposed to
by ntenenz 11y ago
Being computationally limited by GEMM is both a benefit and a pitfall. The operation itself is highly parallelizable and typically compute-bound (as opposed to memory-bound). As a result, advances in GPU performance will translate into improvements in GEMM. However, it also means that algorithmically you're limited by the hardware. GEMM is an _extremely_ mature so algorithmic improvements are unlikely. Instead, any algorithmic improvements would likely stem from replacing GEMM with a more tailored, specialized operation.
- gcr 11y agoDifferent parts of the computation are bound in different ways. The "rule of thumb" for AlexNet-derived networks (with conv layers followed by fully connected layers) is that 95% of the computation time is spent in the conv layers, but 95% of the parameters are stored as the weights of the FC layers. I imagine the later stage could be memory-bound and the former stage is CPU-bound. Not sure if this holds true for other NN structures like GoogLeNet or the weirder recent fully convolutional networks.