cuDNN converts to NHWC for 'tensor core' operations, so if the tensors are already in NHWC you can avoid the conversion. See https://docs.nvidia.com/deeplearni

cuDNN converts to NHWC for 'tensor core' operations, so if the tensors are already in NHWC you can avoid the conversion. See https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#tensor-ops-tensor-transformations-conversion https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-gui... EDIT: I think https://github.com/pytorch/pytorch/issues/19092 https://github.com/pytorch/pytorch/issues/19092 is the RFC for this feature, and it has some more discussion.