3 ms·
I agree this sort of approach looks promising. Maybe using FFTs recurrently to approximate convolutions with input-length filters is the way forward. It's a cle
by cs702 4y ago
I agree this sort of approach looks promising. Maybe using FFTs recurrently to approximate convolutions with input-length filters is the way forward. It's a clever idea. I'm making my way through the paper. Don't fully understand it yet.
The main issue I've seen with other wannabe-sub-quadratic-replacements for self-attention is that they all rely on some kind of low-rank/sparse approximation that in practice renders LLMs incapable of modeling enough pairwise relationships between tokens to achieve state-of-the-art performance.
I'm curious to see if this kind of approach solves the issue.
- inciampati 4y agoDoes FFT not suggest an implicit kind of quantization to the dimension of the time space model? It's something that I have not understood about S4, Hippo, Hyena and other FFT-trick models.
- cs702 4y agoFFT's inputs are discrete to begin with (evenly spaced samples in the time domain), so I kind of see what you're trying to say... but I don't fully understand the work yet. I'm slowly making my way, starting with getting acquainted with classic state-space models. That said, my sense is there's a significant difference between (a) learning to approximate the matrix of n×n interactions with more run-of-the-mill methods, such as some kind of linear decomposition with learned fixed coefficients, versus (b) learning to approximate the matrix of n×n interactions with dynamic methods that "find the most suitable decomposition for each sample on-the-fly," which is what these class of models appears to be doing. Apologies if all this sounds very hand-wavy; it's the best I can do at the moment.