3 ms·
And the other problem is that tensor kernels are just a bear to optimize. Even on the CPU, if you write a 3 loop implementation of matrix multiplication you're
by ChrisRackauckas 7y ago
And the other problem is that tensor kernels are just a bear to optimize. Even on the CPU, if you write a 3 loop implementation of matrix multiplication you're leaving behind a ton of performance. You want to use a blocked algorithm which specifically knows your cache sizes, and it comes out to like a 12 nested loop implementation which can be architecture-dependent and maybe have a tiny bit of assembly magic in the middle. That's just a CPU, and that's just a dense matmul. This is why things like Halide exist for trying to generate optimized kernels to things like stencil operations, since "just write the loop" really isn't a good strategy. Couple that with the fact that you need to specialize it to the architecture, and indeed a lot of it isn't documented, and you have a hard time getting something as fast as CuDNN.
If you haven't seen this stuff, take a look at: https://www.cs.cmu.edu/afs/cs/project/pscico-guyb/realworld/www/slidesF10/FLP99.pdf https://www.cs.cmu.edu/afs/cs/project/pscico-guyb/realworld/... . That's cache-oblivious, so it still leaves some performance on the table (which is why BLAS doesn't use a cache-oblivious form), but gets the point across that you can do a ton of algorithmic work here.
What Julia gives you is something where you can directly write the loops and compile to CUDA kernels, and it will run fast. The issue there isn't making the CUDA kernels, but coming up with the algorithm. CuDNN is proprietary so you can't just copy their tensor kernels into Julia. The Julia Lab has some individuals interested in building tensor compilers for this case though, where you can abstractly define such an operation and it will try to come up with a fast version. And then hopefully build binaries from it so it's "compile once"? But this stuff is just hard and there's no easy answer, though Julia is probably the best approach for building an answer IMO :).