4 ms·

Disclaimer: This used to be my bread and butter, but I'm really rusty after five years of not working on this sort of stuff. That said, after quickly skimming

Disclaimer: This used to be my bread and butter, but I'm really rusty after five years of not working on this sort of stuff. That said, after quickly skimming the example AI-generated kernel I am not seeing anything novel there. While working at nVidia I did see a handful of techniques that, frankly, blew my mind. Thus, I wonder what makes this AI-generated kernel faster than the standard pyTorch kernel, which I presume is simply delegating all the heavy lifting onto cuDNN. My guess, and it's just a guess, is that they are comparing the fastest AI-generated kernel they produced for a very particular set of parameters against whatever kernel cuDNN is picking for that same scenario, and perhaps the subsystem inside cuDNN that picks which kernel to execute out of the very large database it manages chose a suboptimal candidate. Researchers tend to completely ignore this issue and assume that cuDNN is always able to choose the very best kernel in every possible scenario, something that is just not realistic. Maybe there is something else going on, but these sort of "we have beaten this heavily optimized proprietary library" always seem to miss this very important point. Kind regards to any NVidia insiders who may read this. You guys are the brightest people I've ever met.

zahlman 1y ago
> Thus, I wonder what makes this AI-generated kernel faster than the standard pyTorch kernel All of this stuff is way outside my wheelhouse, but maybe "the standard pyTorch kernel" is just a low bar? (https://news.ycombinator.com/item?id=44144346 https://news.ycombinator.com/item?id=44144346)
- david-gpu 1y ago
  Thanks for the link. I did have a quick look at the github repo yesterday, trying to find where this "default pytorch implementation" lives, but it wasn't immediately apparent due to the layers of abstraction in the code. There's little doubt in my mind that the authors of the blog post went after the largest performance delta between their best AI-generated code and the default pytorch implementation, without looking into real-world nuances like "picking a good kernel out of a zillion options" or "production-quality output accuracy". It's cool that this sort of thing is being researched, but the results need to be taken with a grain of salt. Maybe I'm too jaded.

zahlman 1y ago

> Thus, I wonder what makes this AI-generated kernel faster than the standard pyTorch kernel All of this stuff is way outside my wheelhouse, but maybe "the standard pyTorch kernel" is just a low bar? (https://news.ycombinator.com/item?id=44144346 https://news.ycombinator.com/item?id=44144346)

david-gpu 1y ago
Thanks for the link. I did have a quick look at the github repo yesterday, trying to find where this "default pytorch implementation" lives, but it wasn't immediately apparent due to the layers of abstraction in the code. There's little doubt in my mind that the authors of the blog post went after the largest performance delta between their best AI-generated code and the default pytorch implementation, without looking into real-world nuances like "picking a good kernel out of a zillion options" or "production-quality output accuracy". It's cool that this sort of thing is being researched, but the results need to be taken with a grain of salt. Maybe I'm too jaded.