4 ms·
Google's FNet seems to apply FT to NNs? https://arxiv.org/abs/2105.03824 https://arxiv.org/abs/2105.03824
by tracyhenry 5y ago
Google's FNet seems to apply FT to NNs? https://arxiv.org/abs/2105.03824 https://arxiv.org/abs/2105.03824
- sillysaurusx 5y agoThat paper was one of the most misleading papers I've ever seen in NNs. https://twitter.com/theshawwn/status/1393315603973386240 https://twitter.com/theshawwn/status/1393315603973386240 Like many others, I was hyped when it came out. Then it turns out that BERT with half the param count still kicks their butts in accuracy. They also only model the .real component and ignore the .imaginary component entirely, which you can't do and expect good results. But, FFTs are so cool and under-explored that I'm sure they'll be making the rounds in NNs soon. There are lots of advantages to frequency space representations.
- kortex 5y agoI don't understand why the first operations of a CNN are FFT-based decomposition into spatial wavelets. This is basically all the first layers are doing anyways. In fact the filters usually learn both an edge and a centroid at a given orientation. You can get both at the same time with complex wavelets.