In figure 3, it shows that Falcon and Pythia are much less susceptible to the lack of an attention sink than Llama or MPT… seems they work almost as well by jus

In figure 3, it shows that Falcon and Pythia are much less susceptible to the lack of an attention sink than Llama or MPT… seems they work almost as well by just doing naïve windowed attention