3 ms·
It looks like a longformer architecture, but with dedicated "attention sink" tokens at the beginning that provide storage space. Is that right?
by Herring 3y ago
It looks like a longformer architecture, but with dedicated "attention sink" tokens at the beginning that provide storage space. Is that right?