Pytorch relies heavily on the extensive libraries of high-performance kernels provided by NVidia, such as cuDNN. In other words, it goes something like this:

Pytorch relies heavily on the extensive libraries of high-performance kernels provided by NVidia, such as cuDNN. In other words, it goes something like this: Application Pytorch (and similar) cuDNN (and similar) CUDA (and similar) NVidia GPU My opinion, based on what I saw those wizards do, is that reproducing the feature set and efficiency of cuDNN/cuBLAS is deeply nontrivial.