3 ms·

I'm really interested in training NNs in the frequency domain: - a ton of data (JPEGs, MPEGs, MFCCs) exists in the compressed form of FFT-ed features - FFTs a

I'm really interested in training NNs in the frequency domain: - a ton of data (JPEGs, MPEGs, MFCCs) exists in the compressed form of FFT-ed features - FFTs are hardware encoded/decoded & optimised - FFTs are great at separating signal from noise, leading to a huge reduction in input features (eg JPEG encoded) as compared to raw pixel values - convolutions in the time domain are just multiplications in the frequency domain - converting to log polar coordinates turns them into additions - distance between two log polar coordinates could be the loss function However, I've not had much luck getting a frequency domain network like this to converge. Anyone tried something similar with success?

mochomocha 5y ago
I have trained GANs on raw JPEG coefficients with moderate success as a pet project. Read the raw DCT coefficients without decompressing and train in this space. JPEG-decompress the output of the net to reconstruct an image in the pixel space. There are few papers doing similar things for supervised learning tasks iirc
- akrymski 5y ago
  Using a standard loss function like MSE? Yeah it kinda works when you feed JPEG coefficients into a typical time-domain CNN, but mathematically it seems that if you're using frequencies as inputs, your convolutions should become simple multiplications. Am I wrong?
  - mochomocha 5y ago
    Yep, you can successfully do it without convolutions. Here is a pointer if you want to dig deeper: https://eng.uber.com/neural-networks-jpeg/ https://eng.uber.com/neural-networks-jpeg/ (there's prior art to that, but this one is well written)
akrymski 5y ago
I don't see why processing JPEG coefficients directly as input features shouldn't have similar accuracy to a much larger NN using raw pixel data. In particular I'm interested in the efficiency gains from avoiding convolutions and the possibility of running a compressed frequency-domain NN on CPUs.
ashertrockman 5y ago
The choice of activation function isn't entirely clear to me, but I think it's definitely possible to make a network that operates entirely in the frequency domain. It would probably be pretty easy to start experimenting with such a thing with the nice complex number and FFT support in PyTorch 1.8. :) Like you said, there's already a significant connection between convolutional networks and the Fourier domain (the convolution theorem). Tangentially, I've recently worked on a project that focused on implementing convolution in the Fourier domain, and how that allows one to control useful properties of convolutions (like "smoothness" and orthogonality). I made a demonstration of convolution in the Fourier domain in PyTorch, which you might find interesting: https://nbviewer.jupyter.org/github/locuslab/orthogonal-convolutions/blob/main/FFT%20Convolutions.ipynb https://nbviewer.jupyter.org/github/locuslab/orthogonal-conv... More generally, you could look here for more code and the corresponding paper: https://github.com/locuslab/orthogonal-convolutions https://github.com/locuslab/orthogonal-convolutions

mochomocha 5y ago

I have trained GANs on raw JPEG coefficients with moderate success as a pet project. Read the raw DCT coefficients without decompressing and train in this space. JPEG-decompress the output of the net to reconstruct an image in the pixel space. There are few papers doing similar things for supervised learning tasks iirc

akrymski 5y ago
Using a standard loss function like MSE? Yeah it kinda works when you feed JPEG coefficients into a typical time-domain CNN, but mathematically it seems that if you're using frequencies as inputs, your convolutions should become simple multiplications. Am I wrong?
- mochomocha 5y ago
  Yep, you can successfully do it without convolutions. Here is a pointer if you want to dig deeper: https://eng.uber.com/neural-networks-jpeg/ https://eng.uber.com/neural-networks-jpeg/ (there's prior art to that, but this one is well written)

akrymski 5y ago

I don't see why processing JPEG coefficients directly as input features shouldn't have similar accuracy to a much larger NN using raw pixel data. In particular I'm interested in the efficiency gains from avoiding convolutions and the possibility of running a compressed frequency-domain NN on CPUs.

ashertrockman 5y ago

The choice of activation function isn't entirely clear to me, but I think it's definitely possible to make a network that operates entirely in the frequency domain. It would probably be pretty easy to start experimenting with such a thing with the nice complex number and FFT support in PyTorch 1.8. :) Like you said, there's already a significant connection between convolutional networks and the Fourier domain (the convolution theorem). Tangentially, I've recently worked on a project that focused on implementing convolution in the Fourier domain, and how that allows one to control useful properties of convolutions (like "smoothness" and orthogonality). I made a demonstration of convolution in the Fourier domain in PyTorch, which you might find interesting: https://nbviewer.jupyter.org/github/locuslab/orthogonal-convolutions/blob/main/FFT%20Convolutions.ipynb https://nbviewer.jupyter.org/github/locuslab/orthogonal-conv... More generally, you could look here for more code and the corresponding paper: https://github.com/locuslab/orthogonal-convolutions https://github.com/locuslab/orthogonal-convolutions