Ummmm... sure. A layer of a neural network is a linear operator followed by an arbitrary (nonlinear in practice) mapping. Which means that neural networks can t

Ummmm... sure. A layer of a neural network is a linear operator followed by an arbitrary (nonlinear in practice) mapping. Which means that neural networks can trivially implement linear operators (like the Fourier transform) by making the second mapping to be identity. QED