3 ms·
When you say it didn't help, can you clarify what you're measuring? In the context of this post, I think both the performance your task, and the number of outli
by abeppu 3y ago
When you say it didn't help, can you clarify what you're measuring? In the context of this post, I think both the performance your task, and the number of outlier weights (and their magnitude) are important.
- chessgecko 3y agoI was just looking at doing this in pretraining, so I was looking at pretraining losses. The difference was within the range of usual noise so I didn't keep trying.
- lucidrains 3y agothis is fixing a different issue, not the one you are measuring.
- chessgecko 3y agoIt wasn't really the goal of my experiment to fix this issue for sure, I was trying to see if you could improve attention by decoupling the key used by a position for itself and for future tokens. Open to being wrong here, but wouldn't it be functionally similar to adding a constant to the softmax denom? the function could sort of learn a specific position to have sink and q multiply to one, then removing it before multipling with v would be exactly identical?
- waynecochran 3y agoThe question concerns outliers ... how did the change manage them?