3 ms·

Look into RWKV.

• JohannaAlmeida 6mo ago
  Yeah RWKV is definitely related in spirit (recurrent state for long context). Here I’m combining local windowed attention with a gated recurrent path + KV cache compression, so it’s more hybrid than fully replacing attention