You can cache transformers though? Although the cache grows in size as more input tokens are added, while RWKV has to keep it in a single hidden state that's al

You can cache transformers though? Although the cache grows in size as more input tokens are added, while RWKV has to keep it in a single hidden state that's always the same size, but it still speeds up inference. The huggingface transformers library exposes this as past_key_values, here's an example on GPT-2: https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.models.gpt2.modeling_tf_gpt2.TFGPT2DoubleHeadsModelOutput.past_key_values https://huggingface.co/docs/transformers/model_doc/gpt2#tran...