2 ms·
I also saw the author distinguished internal versus output softmax. I think he'd apply his modification only to internal softmax and let the external force an o
by rrobukef 3y ago
I also saw the author distinguished internal versus output softmax. I think he'd apply his modification only to internal softmax and let the external force an output.
- cs702 3y agoYes, it makes sense to apply this only to the Softmax we use to compute attention. It makes no sense to apply it to the output Softmax, which must compute a probability distribution over the vocabulary.