I wonder how much change is needed to how we use GPUs for this to yield real gains. Can we revise libcuDNN (and other higher math libs) only, or CUDA itself?

I wonder how much change is needed to how we use GPUs for this to yield real gains. Can we revise libcuDNN (and other higher math libs) only, or CUDA itself? Or must GPU hardware change too?