I see, the wording confused me. You don't really "put Fourier transforms around the convolutional layers" because you have to completely replace the convolution

I see, the wording confused me. You don't really "put Fourier transforms around the convolutional layers" because you have to completely replace the convolutions. This seems to be done in some cases. I guess it isn't done more widely because the "standard" convolution kernels are very small and the performance would actually be worse?