Another paper related to attention distillation, although doing something far more radical: transformer attention is distilled onto RWKV-like model: https://hug

Another paper related to attention distillation, although doing something far more radical: transformer attention is distilled onto RWKV-like model: https://huggingface.co/papers/2505.03005 https://huggingface.co/papers/2505.03005