6 ms·
The Fourier transformation is a special case of a linear transformation. Linear neural networks can represent any linear transformation. Both can also be expres
by threatripper 5y ago
The Fourier transformation is a special case of a linear transformation. Linear neural networks can represent any linear transformation. Both can also be expressed as matrix multiplication.
It's not really all that surprising if you know some of the math behind neural networks, matrix algebra, linear transformations or the Fourier transformation.
- amelius 5y agoYes but this article shows that the matrix coefficients of the Fourier transform can be learned.
- threatripper 5y agoThis is also not really surprising for a single layer neural network.
- visarga 5y agoIt might not be surprising when they implemented it by supervised learning, but when they also solved it by reconstructing the input it was like a rediscovery.
- madhadron 5y agoThat's because "learning" in neural networks is a fancy way of saying "curve fitting."
- amcoastal 5y agoI dont think its surprising but its definitely a cool little exercise. Somewhat aside of the content of the post, I could see some use cases where specifically implementing DFT layers in your architectures could lead to improved performance over using just the raw inputs or activations. Noisy datasets, or synthetic data based transfer learning come to mind as potential uses for the DFT as a step in your ML pipeline.
- threatripper 5y agoThere is the FFT in TensorFlow already: https://www.tensorflow.org/api_docs/python/tf/signal/fft https://www.tensorflow.org/api_docs/python/tf/signal/fft
- amelius 5y agoAnd FFT (Fast Fourier Transform) is more efficient than matrix multiplication (EDIT: by a matrix with Fourier elements).
- mochomocha 5y agoThis is analogous to saying "an umbrella is better than a toaster". They are two different things.
- ok123456 5y agoYou can implement a DFT with matrix multiplication using the vandermonde matrix with roots of the unitary polynomial as the basis.
- amelius 5y agoNot if you perform these operations in order to obtain a Fourier transform.
- cdavid 5y agoFourier transform, more exactly DFT, is a special case of matrix multiplication, when seen as a linear operator. The fact that you can do operations faster than the usual N*2 matrix / vector operation is due to the specific form of the underlying matrix (Van Der Mont matrix). So they are not that different: the specific structure of the Fourier basis allows for more efficient matrix * vector operation.
- deleted 5y ago[deleted]
- thearn4 5y agoI've come to internalize the Fourier transform as a type of sparse matrix algorithm. I.e. something that implements a particular matrix-vector product without requiring explicit construction of said matrix.
- srean 5y agoAnd pushing the parantheses to factor out common subexpressions. Essentially exploiting the distributive law
- enchiridion 5y agoDo you mean the FFT? I've been trying to wrap my head around the Fourier transform lately, and I can see the matrix connection to an FFT, but not the Fourier transform in general.
- bonoboTP 5y agoThe Fourier transform performs a change of basis into the eigenbasis of convolution. It's linear and so in the discrete case can be expressed as matrix multiplication (every discrete linear operation is expressible as matmul).
- amatic 5y agoOn the other hand, if you don't yet know a lot of NN math, like me, it is super surprising. Definitely a cool way to connect FFT and NNs.