3 ms·
When Fast Fourier Transform Meets Transformer for Image Restoration (2024)
- waynecochran 5mo agoWas there a conclusion?
- gryfft 5mo ago[2024]
- deleted 5mo ago[deleted]
- TimorousBestie 5mo agoThere have been some interesting advances in trying to add spectral information to the data that a learning architecture has at its disposal, but there are a couple roadblocks that I don’t think have been solved yet. 1. Complex-valued NNs are not an easy generalization of real ones. 2. A localization in one domain implies non-local behavior in the other (this is the Fourier uncertainty principle). Fourier Neural Operators (FNOs) come close to what I want to see in this area but since they enforce sparsity in the spectral domain their application is necessarily limited.
- FuckButtons 5mo agoI do wonder if a wavelet transform might be better.
- TimorousBestie 5mo agoI think one can do better with a wavelet, shearlet, or curvelet transform that is adapted to the problem domain at hand. But the uncertainty principle still haunts those transforms, and anyway the goal is to be domain-agile.
- sorenjan 5mo agoSee also: CosAE: Learnable Fourier Series for Image Restoration (2024) https://sifeiliu.net/CosAE-page/ https://sifeiliu.net/CosAE-page/
- jongala 5mo agoRelatedly, Marcin Wichary wrote a nice post about using FFT to remove moiré and halftone effects when scanning images that were printed with halftones. It's from 2021: Moiré no More (https://newsletter.shifthappens.site/archive/moire-no-more/ https://newsletter.shifthappens.site/archive/moire-no-more/).
- krackers 5mo agoI'd like to see a sequel where the fractional fourier transform is used for image restoration