(b) is the sacrifice made in these linear attention type architectures. As a mitigation, you can leave a few normal attention layers in the model but replace t

(b) is the sacrifice made in these linear attention type architectures. As a mitigation, you can leave a few normal attention layers in the model but replace the rest.