After a very quick read, that's my understanding too: It's just KNN search with some bells and whistles. So I agree on points 1-3. When something works well, I

After a very quick read, that's my understanding too: It's just KNN search with some bells and whistles. So I agree on points 1-3. When something works well, I don't care much about point 4. Personally, I've had only mixed success with KNN search on long sequences. Maybe I haven't done it right? I don't know. In my experience, nothing seems to work quite as well as explicit token-token interactions by some form of attention, which as we all know is too costly for long sequences (O(n²)). Lately I've been playing with https://github.com/hazyresearch/safari https://github.com/hazyresearch/safari , which uses a lot less compute and seems promising, though it reminds me of things like FNet. Otherwise, for long sequences I've yet to find something better than https://github.com/HazyResearch/flash-attention https://github.com/HazyResearch/flash-attention for n×n interactions and https://github.com/glassroom/heinsen_routing https://github.com/glassroom/heinsen_routing for n×m interactions. If anyone has other suggestions, I'd love to hear about them.