10 ms·
The FFT Strikes Back: An Efficient Alternative to Self-Attention
- larodi 2y agoMan, this all really gets increasingly more complex in increasingly complex math…
- jampekka 2y agoThis is actually simpler than most self attention methods.
- singularity2001 2y agoDeep learning actually simplifies the extremely complex math of previous machine learning and statistics/stochastics into a very reasonable set of operations: matrix multiplications and some very simple activation functions (plus automatic derivates, some magic and some scientific glasses which you can ignore)
- avereveard 2y agoIsn't flash attention already n log n?
- GaggiX 2y agoNo https://arxiv.org/abs/2402.07443 https://arxiv.org/abs/2402.07443
- DrNosferatu 2y agoOnly in space complexity, not in time complexity.
- pointlessone 2y agoOK, I admit that the math flies way over my head and I barely understand the text around the math. Can someone please explain (in basic English) how this is equivalent to attention mechanism? What friquencies does it talk about? How does it encode positional relations between tokens?
- jampekka 2y agoThe actual mechanism at least is quite simple. Essentially it takes the FFT of the input embeddings, multiplies it elementwise with weights that are gotten from the input embeddings using an MLP (plus a constant (but learnable) bias) and then runs it through an activation function and finally takes the inverse FFT. The "frequencies" are probably something quite abstract. FFT is often used in ways where there aren't really clear frequency interpretation. The use is due to convenient mathematical properties (e.g. the convolution theorem). Rather amazing if this really works well. Very elegant.
- xeonmc 2y agoEssentially leveraging Convolution theorem. Same philosophy pops up in many places, e.g. DFT calculations
- physicsguy 2y agoYep, it's very common to use this. we used it for grid based pairwise interactions since it turns N^2 op into N log N.
- dekhn 2y agoThat sounds just like Particle Mesh Ewald, which we use in molecular dynamics to approximate the forces of pairwise interactions (interpolated on a grid). Ihttps://en.wikipedia.org/wiki/P3M https://en.wikipedia.org/wiki/P3M
- physicsguy 2y agoIt's similar but I worked on magnetic spin systems with dipole-dipole interactions, so there wasn't the interpolation part, and as I understand it in Ewald summation you're always assuming periodic boundary conditions. In our spin systems you basically pre-compute the interaction kernel tensor and can either take into account periodicity or ignore it depending on what sort of system you're looking at. Often you don't want the periodic effect since the dipole-dipole interaction is only one of many, much of the interesting phenomena in magnetics is in the interplay between short range forces and the long range forces. At each time step you FFT to the magnetisation tensor and then multiply with the interaction tensor, then iFFT.
- xeonmc 2y agoBasically leverages convolution theorem[0]: expensive convolutions in direct space becomes simple multiplications in reciprocal space, and vice versa. Whereever you have a convolution operation on your data, transform them to the conjugate domain to turn it into multiplication. In other words, work in the domain that is natural to your data. [0] https://en.wikipedia.org/wiki/Convolution_theorem https://en.wikipedia.org/wiki/Convolution_theorem
- snickmy 2y agothis is a great way to put it, that said, it was not obvious to me that the attention space (how it is structured in LLMs) is a frequency domain
- kkylin 2y agoExactly. Exploiting the structure of the matrix (e.g., it is well approximated by a circulant matrix) is natural if there is structure to exploit. If everything in the preprint holds up, that might suggest some symmetries (e.g., approximate stationarity in time) in the data at hand.
- evanb 2y agoA cartoon: To form a coherent idea you need to coordinate a lot of tokens. In other words, ideas are long-distance correlations between tokens. Ideas are the long-wavelength features of streams of tokens. Is it exactly right? No. But as a cartoon it can motivate exploring an idea like this.
- kridsdale1 2y agoI like this. Anything that connects new synapses in my skull via analogy is a good post.
- spot5010 2y agoRight. This makes sense. But why Fourier space in particular. Why not, for example, a wavelet transform.
- yorwba 2y agoI don't see how you could fit causal masking into this framework without having to do n different FFTs, and there's no mention of positional embeddings either, so I guess the self-attention implementation being compared against is noncausal NoPE, which would make this a case of baseline sandbagging and maybe not so impressive. If the results were close to state-of-the-art, probably the author would've mentioned it?
- TheDudeMan 2y agoThey do show their model as winning every category in Long Range Arena (LRA) benchmark. Hopefully they have not excluded losing categories or better models.
- yorwba 2y agoWinning against their own baseline, not against the current best-performing model. Which apparently is S5 currently https://paperswithcode.com/sota/long-range-modeling-on-lra https://paperswithcode.com/sota/long-range-modeling-on-lra with 87.46 overall vs. 58.31 here.
- yagizdegirmenci 2y agoGoogle introduced this idea in 2022 with "FNet: Mixing Tokens with Fourier Transforms" [0]. Later they found out that, performance of their TPU(s) for matrix multiplication was faster than FFT in the most scenarios. [0]: https://arxiv.org/abs/2105.03824 https://arxiv.org/abs/2105.03824
- frodo8sam 2y agoI would guess that the FFT scales better as you increase the number of tokens in the context window. Interesting Google's models outperform their competitors on context size.
- Daniel_Van_Zant 2y agoI'm glad someone else had the same thought. I have been wondering what their "secret sauce" is for a while given how their model doesn't degrade for long-context nearly as much as other LLMs that are otherwise competitive. It could also just be that they used longer-context training data than anyone else though.
- roflmaostc 2y agoReference for the later part?
- yagizdegirmenci 2y agoThe section "3.3 Implementation" is mostly about hardware level speedups, which basically says: On GPU(s) FFT is consistently faster, but in TPU(s), for shorter sequences matrix multiplication was faster.
- xphos 2y agoYeah but a comparison in power utilization is needed too. You can build hardware that is better than a GPU at something i.e MatMul being really efficient and fast. However, actual FFT hardware would annihilate power and speed at large enough n. Simply because the number of multiplications MatMul does is O(n^3) as opposed to the O(n log n) multiplies that FFT does (complex verse real multiplies with holding).
- cs702 2y agoTL;DR: 1. Take FNet (https://arxiv.org/abs/2105.03824 https://arxiv.org/abs/2105.03824). 2. Replace the fixed (frequency-domain) convolution filter with one that is dynamically computed from the data. 3. Apply non-linear functions to both real and imaginary components, before mapping the convolved data back to the time domain.
- A7C3D5 2y agoI'll never not read FFT as Final Fantasy Tactics.
- seangrogg 2y agoI came here just to see if anyone else shared this same experience. HN did not disappoint.
- deleted 2y ago[deleted]
- DrNosferatu 2y agoCan someone confirm the big O time complexities of 1. traditional Self-Attention; 2. Flash-Attention? 3. Any novel others?
- DrNosferatu 2y agoOk, what FlashAttention changes is space complexity: from O(N^2) to O(N). Time complexity is still ~O(N^2) as with standard Self-Attention. In other words, optimizes practical runtime through I/O reduction without altering asymptotic complexity
- TheDudeMan 2y agoMamba is O(n). But I guess it has other drawbacks.
- DrNosferatu 2y agoYes, in Mamba accuracy seems to goes down and has trouble in exact token recall. But, I would say it might be good for very power efficient edge deployment, and ultra long contexts.
- imtringued 2y agoMamba is solving a different problem than transformers. What Mamba does is take an initial state s_0 and an input u_0, to produce a new state s_1 and an output o_1. It's basically modeling a very complicated state machine. I can easily think of half a dozen applications where this is exactly what you want and it is better than transformers, but LLMs are not among them. Essentially most control problems boil down to what Mamba does. In fact, I would say that Mamba as an architecture is probably the non-plus ultra for modeling mechanical system dynamics.
- kla-s 2y agoActually its a little more nuanced: Operation Type|Mamba Complexity|Transformer Complexity Training(per iteration)|O(L)|O(L^2) Autoregressive Inference(per step)|O(T)|O(L) Memory Requirements|O(C)|O(L) Where: L stands for the sequence length. T denotes a fixed constant that accounts for compression and selection time in Mamba's autoregressive inference. C reflects the fixed size of the SSM (State Space Model) latent state in Mamba Per: https://github.com/state-spaces/mamba/issues/196 https://github.com/state-spaces/mamba/issues/196
- deleted 2y ago[deleted]
- farhanhubble 2y agoMakes me think what if NNs are treated as black box signal processing units. What other techniques can we borrow from signal processing?
- pillefitz 2y agoIn image processing at least, NN typically learn a Fourier or Wavelet representation in their first layers. Biggest benefit of applying a transformation beforehands is to reduce training time / obtain better generalization by "removing the dimension that doesn't matter". E.g. in a suitable space, one coordinate could represent the rotation of an object. You could do the transform and discard this dimension if your NN should be rotating invariant.
- SJC_Hacker 2y agoIn image processing I thought there was a whole host of specialized algorithms, such as edge detection, SCC, etc. that were run before the data was even fed into the ANN.
- bizarrevr 2y agoI am very out of my depth here haha with the math, appreciate those below taking the time to baby walk me along with the understanding! Great links too!
- TheDudeMan 2y agoMachines going to be thinking in the frequency domain. We cooked.
- TeMPOraL 2y agoOur bodies have parts that do processing in frequency domain. Who knows, maybe it'll turn out we are thinking in the frequency domain, too.
- SJC_Hacker 2y agoYes, the cochlea for one basically performs an FT on the incoming input signal, at least with respect to magnitude. The phase portion is still in the time domain
- DrNosferatu 2y agoWell done! Besides immediate speed gains, I guess this opens the door to ultra-long contexts. Larger than say, 16M tokens.
- ipunchghosts 2y agoThis guy's has had a lot bangers lately.
- ipunchghosts 2y agoOverall, i like this guys papers but they strike me as someone who is very smart but hasnt looked through the literature carefully. Many of the techniques he is proposing were already done about 5-6 years ago. However, I imagine that because the field is flooded with new humans, they are not aware of this research or think it will lead to a fruitful end (which many other researchers have already thought of this and it didn't lead anywhere hence why it was abandoned). Overall, it seems we are starting to recycle ideas because there isnt enough lit review and or mentoring from senior deep learning / ML folks who can quickly look at a paper and tell the author where the work has been already investigated.
- liamdgray 2y ago> Overall, it seems we are starting to recycle ideas because there isnt enough lit review and or mentoring from senior deep learning / ML folks who can quickly look at a paper and tell the author where the work has been already investigated. Arguably, the literature synthesis and knowledge discovery problem has been overwhelming in many fields for a long time; but I wonder if, in ML lately, an accelerated (if not frantic) level of competition may be working against the collegial spirit.
- ipunchghosts 2y agoI think it's been accelerated by the review community being overwelmed and the lack of experienced researchers with the combination of classic ML, deep learning, transformers, and DSP backgrounds -- a rare breed but sorely needed.
- gessha 2y agoReviving old ideas and comparing them to SOTA is not necessarily bad especially if they provide benefits over the SOTA model. It brings the old ideas into the community idea cache if you will. It’s somewhat annoying if the authors do it thinking it’s a novel idea when it fact it’s a 20-30 year old one. This reminds me of some HN comments about rocketry ideas and in the thread one of the comments was “Everything in rocket science has been theorized/tried by some Russian scientist 40-50 years ago” and it still gives me a chuckle.
- markisus 2y agoThe Fourier transform is taken along the “token” dimension. However, in many applications, this dimension is not meaningful. That’s why transformers are a great option for consuming data which is permutation invariant. I would like to see additional experiments using the lesser known Fourier transform over finite groups [1], which is permutation invariant but shares many properties with the standard Fourier transform. I also wonder if this becomes the next big thing for LLMs, how easy will it be for inference engines(eg vLLM, llama.cpp) to integrate it? [1] https://en.wikipedia.org/wiki/Fourier_transform_on_finite_groups https://en.wikipedia.org/wiki/Fourier_transform_on_finite_gr...
- Y_Y 2y agoWhat's the finite group in this case?
- markisus 2y agoI’m thinking the integers mod 2^n where n is something computers are good at (8, 32, 64). You have hardware support the group operation.
- yorwba 2y agoThat is the traditional Fourier transform, except it can be a cyclic group of any size, doesn't need to be a power of 2. (Though FFTs with 2^n input size are particularly easy to implement.) And it's not permutation invariant.
- markisus 2y agoI was careless in my thinking, thanks for the correction. I was imagining since you sum the group elements in any order, there is a permutation invariance. But the group elements themselves play the role of the "token index" and group elements are not interchangeable. To actually make this idea interesting, one would have to use a group in which the choice of group element assigned to each input token would not affect the result.
- 2y ago
- antoineMoPa 2y agoWhat I don't get about attention is why it would be necessary when a fully connected layer can also "attend" to all of the input. With very small datasets (think 0 - 500 tokens), I found that attention makes training longer and results worse. I guess the benefits show up with much larger datasets. Note that I'm an AI noob just doing some personal AI projects, so I'm not exactly a reference.
- DrVoidberg 2y agoattention grows dynamically with the input size - mlps not
- andersource 2y agoA fully connected layer has different weights for each feature (or position in input in your formulation). So the word "hello" would be treated completely differently if it were to appear in position 15 vs. 16, for example. Attention, by contrast, would treat those two occurrences similarly, with the only difference depending on positional encoding - so you can learn generalized patterns more easily.
- antoineMoPa 2y agoI think that this is the explanation I needed, thanks!
- smallmancontrov 2y agoThis is the case with most clever neural architectures: in theory, you could always replace them with dense layers that would perform better with enough resources/training, but that's just it, efficiency matters (number of parameters, training data, training time, FLOPS) and dense layers aren't as efficient (to put it mildly). You have seen this play out on a small scale, but if you calculate the size of the dense layers necessary to even theoretically replicate a big attention layer or even convolution, to say nothing of the data needed to train them without the help of the architecture's inductive bias, you will see that the clever architectures are quite necessary at scale.
- deleted 2y ago
- pk-protect-ai 2y agoDo I get it wrong, that this will be incompatible with OrthoGrad¹ optimizer? [¹] https://arxiv.org/abs/2501.04697 https://arxiv.org/abs/2501.04697
- yorwba 2y agoThese seem like orthogonal developments to me that could easily be combined. What made you think they might be incompatible?
- soulofmischief 2y agoAmazing. I had a similar insight and did exactly this some time ago, transforming the input into the frequency domain in order to preserve input energy and encode information in the complex space, plus filtering based on global context (different filter implementation than discussed in this paper) but only trained the model locally and didn't bother seeing how it scaled. Congrats on the hard work!
- EGreg 2y agoBut does it allow unlimited context windows, like RoPE does, which LLaMa 2 now features?
- leecarraher 2y agothis seems to follow the similar FNO work by nvidia, and switching to frequency domain is usually in any computer scientist's toolbox at this point, however, I'm curious if this translates to real gains for real architectures. FFT makes use of imaginary numbers to encode the harmonics of the signal, these are generally not amenable to gpu architectures. Would fast walsh hadamard suffice? Sometimes the 'signal mixing' is more important than the harmonics of a compositions of sines. Or do we go further down the rabbit hole of trained transformation and try out wavelets? I am an avid FFT fan, (love fast johnson lindenstrauss transform using the embedded uncertainty principle for RIP), but sometimes real hardware and good theory dont always align (eg there are sub ternary matrix multiplies, but they are rarely used in DL)
- hansvm 2y agoComplex numbers work just fine on a GPU. You just represent the data as a real-part matrix and an imaginary-part matrix and do the (ac-bd)+(ad+bc)i stuff in matrix land instead of on complex scalars.
- Y_Y 2y agoWhat's wrong with complex numbers on GPUs? You don't have to do anything special. It's obviously faster if you can make simplifying assumptions like "the input signal is purely real" but otherwise at worst you're dealing with pairs of reals (or floats) and don't have to think about philosophical implications. https://docs.nvidia.com/cuda/cufft/ https://docs.nvidia.com/cuda/cufft/
- leecarraher 2y agogpus dont implement complex number fp math, you have to bolt it on as extra logic. cufft works because you can recursively predict the imaginary and real component paths in the butterfly network. between layers you have fft->ifft , is this cost memory locality-wise worth it, or is it better to find ways to tamp down n in n^2 self attention by windowing, batching, gating, many other solutions. im not saying this work isn't cool, FNOs are really cool especially for solving PINNs and related continuous problems, are llms continuous problems, does n have to span the entire context window? I'll probably end up experimenting with this as theyve made the code available, but sometimes good theory is good theory, but not necessarily practical.
- wafngar 2y agoShould be a relevant reference: https://arxiv.org/abs/2111.13587 https://arxiv.org/abs/2111.13587 Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, Bryan Catanzaro
- tonetegeatinst 2y agoI sort of grasp big O notation...but this is sort of over my head like most stuff that has to do with computer or electrical engineering. As someone who is absolutely terrible at math, I envy the people who grasp or at least can learn this type of stuff and get an engineering degree and license. All I really know about FFT is that is changes a signal, its somehow used in processing signals of some kind, and it apparently from what I heard was the key to detecting nuclear detonations back in the day.
- dilawar 2y agohttps://m.youtube.com/watch?v=spUNpyF58BY https://m.youtube.com/watch?v=spUNpyF58BY
- MajimasEyepatch 2y agoHaving a decent intuitive notion of Fourier transforms is an incredibly useful tool in your toolbox, even if you can't derive a Fourier transform by hand or write a fast Fourier transform (FFT) algorithm. The basic idea is this: (almost) any (useful) signal can be represented as a sum of sine waves with different frequencies and phases. For example, an electrical signal or a sound wave is a one-dimensional signal where the x-axis is time. This might look like a really complex squiggly line that's hard to work with. Using a Fourier transform, you can separate the individual frequencies of that time-based signal. Then, you can modify the specific frequencies however you want. For example, if you have a lot of random, spiky noise in the signal, that will show up as high frequencies. To clean it up, just do a Fourier transform, throw out any data with a frequency above a certain threshold, and then run an inverse Fourier transform on the remaining data to get back a smoother version of the original signal. This is called a low-pass filter, and it's more or less equivalent to taking a moving average of the original signal. Where it gets really fun is that you can extend this, in a pretty straightforward way, to higher dimensions. A two-dimensional signal, where both the x- and y-axes are space, is just an image. JPEG compression is based on this concept: it removes the high-frequency signal in the image in order to store the data in a more compact form, at the expense of losing some fine detail (or creating those ring-like artifacts, if you throw out too much data). Add a third dimension for time, and now you have video. And so on. The nice thing about all this is that it's very visual, so you can get a good intuition for it without having to know all the math inside and out. Here's a good page with lots of visualizations and interactive examples: https://www.jezzamon.com/fourier/index.html https://www.jezzamon.com/fourier/index.html And this 3Blue1Brown video does a good job of explaining it: https://youtu.be/spUNpyF58BY?si=dz0z-s8NftW3Htun https://youtu.be/spUNpyF58BY?si=dz0z-s8NftW3Htun
- mkw5053 2y agoI'm interested in how this would work for generative models. It's not obvious how you'd implement causal masking in the frequency domain. And the modReLU activation seems critical but adds implementation complexity. Would love to see how this scales on truly massive context lengths where the theoretical advantages should really shine.
- bjenik 2y agoWorking in the Fourier domain has been a staple of scientific and engineering applications. Learning those interactions rather than just hardcoding them has been fairly widely explored as well - the term to look for is Fourier Neural Operators [1][2]. It turns out you can prove universality even if the nonlinearity remains in the real domain [3]. The concept is fairly mainstream nowadays, to the degree that Jensen talked about it in his GTC keynote in 2021 [4] and there’s even a mainstage TED talk about its applications [5]. A nice property of doing things this way is that your model ends up being resolution-invariant which is particularly interesting for engineering domains. Scaling these methods has sparked the "let’s do a fully deep-learning-based weather model"-race [6][7]. As for using this on text data: my intuition would be that is going to not work as well because of a fairly unique property of text: for image, video and scientific data each individual element is of approximately equal importance, whereas in text you can have discrete tokens like a "not" somewhere in there that change the meaning of everything around it fairly significantly and you’d want that all to all interaction to capture that. Any kind of mixing that smoothes things out is going to inherently be at a disadvantage - probably true to some degree for most of those efficiency saving methods and why we’re seeing more limited adoption on text. [1] https://arxiv.org/abs/2010.08895 https://arxiv.org/abs/2010.08895 [2] https://www.nature.com/articles/s42254-024-00712-5 https://www.nature.com/articles/s42254-024-00712-5 [3] https://jmlr.org/papers/v22/21-0806.html https://jmlr.org/papers/v22/21-0806.html [4] https://www.youtube.com/watch?v=jhDiaUL_RaM&t=2472s https://www.youtube.com/watch?v=jhDiaUL_RaM&t=2472s [5] https://www.ted.com/talks/anima_anandkumar_ai_that_connects_the_digital_and_physical_worlds https://www.ted.com/talks/anima_anandkumar_ai_that_connects_... [6] https://arxiv.org/abs/2202.11214 https://arxiv.org/abs/2202.11214 (Feb 2022) [7] https://www.wired.com/story/ai-hurricane-predictions-are-storming-the-world-of-weather-forecasting/ https://www.wired.com/story/ai-hurricane-predictions-are-sto...
- johntb86 2y agoDoes anyone have an intuition about why looking at things in the frequency domain is helpful here? The DC term I can see, but I wouldn't expect the input data is periodic enough that other frequencies would be meaningful.
- quantadev 2y agoSince I believe consciousness itself is made of EMF waves, generated by neural activity (rather than synaptic firings themselves, which I view merely as signal carriers like the I/O to/from brains), I'm glad to see it any time FFTs are used in any way in NNs or AI research. I started to develop my own custom type of MLP (multilayer perceptron), that was going to use frequencies and phase angles (FFT) as the "model weights", but then I decided probably it would only outperform the standard MLP if the training data itself was periodic in nature, rather than with language tokens or even image data. Not sure if that's correct or not since Fourier Series shows us ANY arbitrary function can be simulated via a superposition of waves. I still believe if we do achieve something amazing (i.e. competitive with SOTA AI models) with a wave-based NN, it won't create any 'true' qualia however, because simulating EMF waves in a computer is not the same as real EMF waves existing. I think even a 100% perfect simulation of a brain in a computer, for example, will always be a 'zombie' (no qualia). This is obvious if consciousness is indeed made of waves; but it's astounding how few NN-researchers seem to be so illiterate in the field of neuroscience that they don't realize how much evidence there is that consciousness is a wave phenomena.
- semiinfinitely 2y agoI see no mention of prior work Hyena Operator which already demonstrated O(n log n) full context mixing several years ago. https://arxiv.org/abs/2302.10866 https://arxiv.org/abs/2302.10866
- intalentive 2y agoHyena came out of Albert Gu's prior work in the same lab. https://arxiv.org/abs/2111.00396 https://arxiv.org/abs/2111.00396
- hinkley 2y agoI think in the age of telemetry we are also missing a substantial trick by not applying FFTs to cloud telemetry to smoke out epicycles and metastable systems before rather than after they trigger drama. This is unfortunately within my level of notice but not within my level of skill, and my dance card is already full. "SLA's are most likely to be violated 23-25 minutes after a service deployment. Hmm, I wonder why that is... Oh no."
- mtrovo 2y ago"I'm afraid I cannot deploy your application, Dave" Jokes aside one area this could be really worth money is predicting cycles of traffic and saving with ramp up and ramp down of server instances. It's the kind of work that if you're doing it out of your time the company would never give you greenlight it but if you pack it as a shelf product they would totally buy it.
- hinkley 2y agoYep. I worked on a SaaS for one particular retail industry, and I would bet you anything they see more traffic when biweekly paydays and the calendar line up in certain ways. Either a couple days before (window shopping) or a few days after, once other bills are paid. That extra half paycheck when a month has 5 Mondays or Fridays in it...
- gunian 2y agoif simping is bad doesnt that mean polynomial time complexity would be the best?
- 29athrowaway 2y agoIs this why biological neural networks are spiking?
- gotoeleven 2y agoAt a high level, it's very unintuitive to me that transforming to the frequency domain for analyzing long sequences of tokens--when there is in general no expectation that the sequences have a periodic structure--can improve efficiency. Stated another way, how can it be possible that it is more efficient to translate the sequence into a series of N variables, where the nth variable is the sum of every nth term of the sequence, if it is unlikely that any relationship between these variables holds for any fixed period? If I combine the 1st 4th 7th 10th .... elements of the sequence, how do we expect the addition of anything beyond the first two elements to add anything but noise? Stated another another way, if I'm going to approximate a function as a sum of sine waves, this is most efficient when the function is periodic and requires more and more sine waves in the sum to approximate the function on a larger and larger domain when the function is not periodic.
- fithisux 2y agoVery interesting idea.
- a-dub 2y agohm. nothing that happens in the fourier domain can touch the phases, which seems like a constraint that would change the behavior of other layers. the default bias of -0.1 with relus and what i would expect to be a flattish spectrum also seems like it would make for a sparse representation in the fourier domain. i assume this is learning the text embeddings at training time, if so, i'd be curious how the constraints of going through the fft and filtering magnitudes would/could change how the embeddings look.
- nialv7 2y agoHow does this compare with Mamba? Intuitively it feels like this should have similar performance to Mamba, but the math is a lot simpler and easier to understand.
- whoisthemachine 2y agoMy EE education keeps coming back.
- deleted 2y ago[deleted]
- alexkranias 2y agoIs this not just a global convolution?