4 ms·

I thought that - like numpy - pytorch was a thin wrapper over a C++ library. So why the slowdown?

w-m 5y ago
...a C++ library with a CUDA backend. But these high-performance building blocks might only be saturating the GPU fully if the data is large enough. I haven't looked at implementing these things, but I imagine uf you have smaller networks and thus less data, the large building blocks may not be optimal. You may for example want to fuse some operations to reduce memory latency from repeated memory access. In PyTorch world, there are approaches for small networks as well, there is https://github.com/NVlabs/tiny-cuda-nn https://github.com/NVlabs/tiny-cuda-nn - as far as I understand from the first link in the README, it makes clever use of the CUDA shared memory, which can hold all the weights of a tiny network (but not larger ones).
- rejuvyesh 5y ago
  Not related to the authors and don't have the same machine, but on a V100, tiny-cuda-nn performance for the blog post matrix power example: Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Initial Train Loss: 5.7188 Initial Test Loss: 5.2812 Took: 11.41 seconds Train Loss: 0.0354 Test Loss: 0.0514 Took: 11.58 seconds Train Loss: 0.0327 Test Loss: 0.0511 Took: 11.42 seconds Train Loss: 0.0316 Test Loss: 0.0505 I think almost of the time here is python overhead because if we increase the batch size 10x, it still takes the same time: Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Initial Train Loss: 5.5391 Initial Test Loss: 5.5938 Took: 11.03 seconds Train Loss: 0.0444 Test Loss: 0.0545 Took: 11.16 seconds Train Loss: 0.0388 Test Loss: 0.0496 Took: 11.01 seconds Train Loss: 0.0384 Test Loss: 0.0490 See [gist](https://gist.github.com/rejuvyesh/6c428ea12154edbb36cd4359fa75e4c9 https://gist.github.com/rejuvyesh/6c428ea12154edbb36cd4359fa...) for the implementation.
hansvm 5y ago
FFI and allocation overhead, plus not being able to perform global optimizations. Each numpy call is pretty fast, and if that's all you're doing then it's probably pretty close to optimal (ignoring for the moment that their routines explicitly optimize large inputs at the expense of small -- for small vectors you can beat numpy even in pure python if you don't explicitly rewrite the problem as a function of larger vectors with appropriate shapes), but a composite call graph potentially (and often) might have additional optimizations that can be applied. For an illustrative example, imagine computing the sum of all elements of some `x-1` where x is some numpy vector. Each individual operation is dispatched to numpy without any global context, so it must explicitly allocate a buffer for the x-1 result before computing the sum. The astute observer might note that you can instead use np.subtract(x, 1, out=x) or np.subtract(x, 1, out=buffer) to reduce the impact of allocation, but the problem is _still_ dominated by memory bandwidth on modern hardware (at one point in the past memory access was about as slow as a computation, and now it's 100x slower at least), and having to iterate over roughly the same elements twice rather than doing a bunch of CPU/GPU operations at each element roughly doubles the total computation time if everything doesn't fit an a very fast cache. Even when using numpy to its fullest (which most people don't), the model of stringing together a bunch of local optimizations can't always and doesn't usually compete with the model of at least attempting to globally optimize the whole computation.
adgjlsfhk1 5y ago
In general, numpy has a minimum execution time of around 1 microsecond per operation. If your operations are on small arrays, Julia will be much faster since small operations can be very fast (eg adding 2 100 element arrays takes about 10 ns).

w-m 5y ago

...a C++ library with a CUDA backend. But these high-performance building blocks might only be saturating the GPU fully if the data is large enough. I haven't looked at implementing these things, but I imagine uf you have smaller networks and thus less data, the large building blocks may not be optimal. You may for example want to fuse some operations to reduce memory latency from repeated memory access. In PyTorch world, there are approaches for small networks as well, there is https://github.com/NVlabs/tiny-cuda-nn https://github.com/NVlabs/tiny-cuda-nn - as far as I understand from the first link in the README, it makes clever use of the CUDA shared memory, which can hold all the weights of a tiny network (but not larger ones).

rejuvyesh 5y ago
Not related to the authors and don't have the same machine, but on a V100, tiny-cuda-nn performance for the blog post matrix power example: Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Initial Train Loss: 5.7188 Initial Test Loss: 5.2812 Took: 11.41 seconds Train Loss: 0.0354 Test Loss: 0.0514 Took: 11.58 seconds Train Loss: 0.0327 Test Loss: 0.0511 Took: 11.42 seconds Train Loss: 0.0316 Test Loss: 0.0505 I think almost of the time here is python overhead because if we increase the batch size 10x, it still takes the same time: Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Initial Train Loss: 5.5391 Initial Test Loss: 5.5938 Took: 11.03 seconds Train Loss: 0.0444 Test Loss: 0.0545 Took: 11.16 seconds Train Loss: 0.0388 Test Loss: 0.0496 Took: 11.01 seconds Train Loss: 0.0384 Test Loss: 0.0490 See [gist](https://gist.github.com/rejuvyesh/6c428ea12154edbb36cd4359fa75e4c9 https://gist.github.com/rejuvyesh/6c428ea12154edbb36cd4359fa...) for the implementation.

hansvm 5y ago

FFI and allocation overhead, plus not being able to perform global optimizations. Each numpy call is pretty fast, and if that's all you're doing then it's probably pretty close to optimal (ignoring for the moment that their routines explicitly optimize large inputs at the expense of small -- for small vectors you can beat numpy even in pure python if you don't explicitly rewrite the problem as a function of larger vectors with appropriate shapes), but a composite call graph potentially (and often) might have additional optimizations that can be applied. For an illustrative example, imagine computing the sum of all elements of some `x-1` where x is some numpy vector. Each individual operation is dispatched to numpy without any global context, so it must explicitly allocate a buffer for the x-1 result before computing the sum. The astute observer might note that you can instead use np.subtract(x, 1, out=x) or np.subtract(x, 1, out=buffer) to reduce the impact of allocation, but the problem is _still_ dominated by memory bandwidth on modern hardware (at one point in the past memory access was about as slow as a computation, and now it's 100x slower at least), and having to iterate over roughly the same elements twice rather than doing a bunch of CPU/GPU operations at each element roughly doubles the total computation time if everything doesn't fit an a very fast cache. Even when using numpy to its fullest (which most people don't), the model of stringing together a bunch of local optimizations can't always and doesn't usually compete with the model of at least attempting to globally optimize the whole computation.

adgjlsfhk1 5y ago

In general, numpy has a minimum execution time of around 1 microsecond per operation. If your operations are on small arrays, Julia will be much faster since small operations can be very fast (eg adding 2 100 element arrays takes about 10 ns).