4 ms·
> Matrix multiplication is an embarrassingly parallel task. (GPU programmer take). This is a common misconception. In reality, matrix multiplication is _extrem
by banachtarski 6y ago
> Matrix multiplication is an embarrassingly parallel task.
(GPU programmer take). This is a common misconception. In reality, matrix multiplication is _extremely tricky_ to implement a fast parallel algorithm for because the number of useful instructions you can perform per byte is limited and you end up getting memory bound very quickly, so you start needing to think about properly employing LDS, managing bank conflicts, etc. For many applications that actually need to run fast, matrices are great for sorting out the theory, but then it's a good time to come up with something that will actually run fast after the paper problem is sorted out.
- stingraycharles 6y agoIs this why my memory consumption explodes when I increase the batch size of my inputs? Sorry I’m a relative GPU noob, but I find it fascinating to observe some higher level interactions with eg pytorch; it seems like the GPU is a very leaky distraction, and you quickly find yourself battling GPU memory management, truncating input sizes and whatnot in order to deal with the insane memory requirements.
- banachtarski 6y agoBatch size is an orthogonal concept. The reason your memory increases so much is because with larger batches, you need to store more intermediate results before running your optimizer. This would be true with a pure CPU implementation of an SGD (stochastic gradient descent) optimizer as well.