I mean that's pretty much self attention mechanism right? It's just beyond playing with more heads, specialised heads, kv caching etc it doesn't seem like anyb

I mean that's pretty much self attention mechanism right? It's just beyond playing with more heads, specialised heads, kv caching etc it doesn't seem like anybody's figured out the next step here yet. Attention is already pretty atrocious perf even with caching so additional context metadata would have to be implemented carefully.