4 ms·
this is fixing a different issue, not the one you are measuring.
by lucidrains 3y ago
this is fixing a different issue, not the one you are measuring.
- chessgecko 3y agoIt wasn't really the goal of my experiment to fix this issue for sure, I was trying to see if you could improve attention by decoupling the key used by a position for itself and for future tokens. Open to being wrong here, but wouldn't it be functionally similar to adding a constant to the softmax denom? the function could sort of learn a specific position to have sink and q multiply to one, then removing it before multipling with v would be exactly identical?