Well, this looks promising. The key idea is to collect a different set of tokens, with different levels of sparsity for each head, apply regular (dense) self-at

Well, this looks promising. The key idea is to collect a different set of tokens, with different levels of sparsity for each head, apply regular (dense) self-attention over all heads, weighted by pairwise distance, and spread and add the output residuals to their corresponding location in the original sequence. It seems to work really well, judging by the perplexity scores shown in the paper -- though we don't yet know if those perplexity scores will translate into good performance on real-world tasks. I'm going to take a closer look.