Thanks for your answer ! So in the 32k global layer, every token attends to each of the other 32k tokens ? [Edit: You answered the question when you said that

Thanks for your answer ! So in the 32k global layer, every token attends to each of the other 32k tokens ? [Edit: You answered the question when you said that individual attention layers are always dense.]