This paper combines knowledge distillation and efficient attention mechanisms. => It works (still efficient, lower cost). Not an unexpected result, but to the

This paper combines knowledge distillation and efficient attention mechanisms. => It works (still efficient, lower cost). Not an unexpected result, but to their credit, they established a new benchmark to test these combinations. KD+LongFormer is one of the best ones, retaining 95.9% of the performance for 50.7% of the cost.