4 ms·

Neural networks can learn many things but I can't imagine a practical use-case of this.

by ellisv 5y ago

killerstorm 5y ago
I guess it is mostly an illustration that time-domain => frequency-domain transformation is trivial for ANNs. And e.g. it won't make any sense to pre-process your signal with FFT before training ANN since it will do it itself if needed.
- raphaelj 5y ago
  Well, you can still probably reduce the training computational cost by feeding the additional FFT features to your NN.
  - ellisv 5y ago
    The Fourier transform layer might not be the immediate hidden layer.
- ellisv 5y ago
  That's a fair point. My assumption was that the neural network isn't going to be as efficient at the FFT but the article doesn't include that type of assessment.
  - killerstorm 5y ago
    NN is doing matrix multiplications anyway. That's what it does, basically. The property of matrix multiplications is that they are composable, i.e. `X * (Y * z) = (X * Y) * z` that is, in the end you only need one matrix. So what this means in practice is that you have FFT for free. NN is doing a matrix multiplication anyway. Discrete time Fourier Transform is a matrix multiplication. Thus it can simply fold DTFT together with whatever other transform it is doing - it doesn't cost anything.
SuchAnonMuchWow 5y ago
Here is an alternative idea around this: Winograd transform can be seen as a depthwise convolution with fixed weights. And similarly to DTF, winograd transform can be used to speed up convolutions: a 3x3 convolution can be implemented with a winograd transform + a 1x1 convolution + inverse winograd transform. This basically reduce the number of operations performed by 2. Now, if a neural network is able to learn the weights of a depthwise convolution to match the coefficients of a winograd transform, that means you should be able to do this transformation when describing the architecture of your network. That way, you don't rely on your deep learning framework to do the transformation for you. And more importantly you might end up with something more generic: perhaps the weights in the depthwise convolutions will not converge to the weights of a winograd transform, but to weight better suited to what you are trying to learn. This is the same ideas which lead to the development of CNN: replace the fixed weights of convolutions used in traditional computer vision algorithm with weights that are learned by the network.

Neural networks can learn many things but I can't imagine a practical use-case of this.

by ellisv 5y ago

Neural networks can learn many things but I can't imagine a practical use-case of this.

I guess it is mostly an illustration that time-domain => frequency-domain transformation is trivial for ANNs. And e.g. it won't make any sense to pre-process your signal with FFT before training ANN since it will do it itself if needed.

raphaelj 5y ago
Well, you can still probably reduce the training computational cost by feeding the additional FFT features to your NN.
- ellisv 5y ago
  The Fourier transform layer might not be the immediate hidden layer.
ellisv 5y ago
That's a fair point. My assumption was that the neural network isn't going to be as efficient at the FFT but the article doesn't include that type of assessment.
- killerstorm 5y ago
  NN is doing matrix multiplications anyway. That's what it does, basically. The property of matrix multiplications is that they are composable, i.e. `X * (Y * z) = (X * Y) * z` that is, in the end you only need one matrix. So what this means in practice is that you have FFT for free. NN is doing a matrix multiplication anyway. Discrete time Fourier Transform is a matrix multiplication. Thus it can simply fold DTFT together with whatever other transform it is doing - it doesn't cost anything.

SuchAnonMuchWow 5y ago

Here is an alternative idea around this: Winograd transform can be seen as a depthwise convolution with fixed weights. And similarly to DTF, winograd transform can be used to speed up convolutions: a 3x3 convolution can be implemented with a winograd transform + a 1x1 convolution + inverse winograd transform. This basically reduce the number of operations performed by 2. Now, if a neural network is able to learn the weights of a depthwise convolution to match the coefficients of a winograd transform, that means you should be able to do this transformation when describing the architecture of your network. That way, you don't rely on your deep learning framework to do the transformation for you. And more importantly you might end up with something more generic: perhaps the weights in the depthwise convolutions will not converge to the weights of a winograd transform, but to weight better suited to what you are trying to learn. This is the same ideas which lead to the development of CNN: replace the fixed weights of convolutions used in traditional computer vision algorithm with weights that are learned by the network.