3 ms·
Does FFT not suggest an implicit kind of quantization to the dimension of the time space model? It's something that I have not understood about S4, Hippo, Hyena
by inciampati 4y ago
Does FFT not suggest an implicit kind of quantization to the dimension of the time space model? It's something that I have not understood about S4, Hippo, Hyena and other FFT-trick models.
- cs702 4y agoFFT's inputs are discrete to begin with (evenly spaced samples in the time domain), so I kind of see what you're trying to say... but I don't fully understand the work yet. I'm slowly making my way, starting with getting acquainted with classic state-space models. That said, my sense is there's a significant difference between (a) learning to approximate the matrix of n×n interactions with more run-of-the-mill methods, such as some kind of linear decomposition with learned fixed coefficients, versus (b) learning to approximate the matrix of n×n interactions with dynamic methods that "find the most suitable decomposition for each sample on-the-fly," which is what these class of models appears to be doing. Apologies if all this sounds very hand-wavy; it's the best I can do at the moment.