2 ms·
What's wrong with unscaled matrix multiplication? Softmax has some kind of intuition in the context, but why not layer norm or something else instead (if anythi
by lostmsu 3y ago
What's wrong with unscaled matrix multiplication? Softmax has some kind of intuition in the context, but why not layer norm or something else instead (if anything is needed at all)?
- sbszllr 3y agoThe family of sigmoid functions has nice gradient properties with theoretical backing. Good starting read: https://stats.stackexchange.com/questions/162988/why-sigmoid-function-instead-of-anything-else/318209#318209 https://stats.stackexchange.com/questions/162988/why-sigmoid...
- lostmsu 3y agoDoesn't really explain why not more typical ReLU + layer norm or some alternative.