3 ms·
Rotary Embeddings: A Relative Revolution
- PaulHoule 5y agoNo wonder. There is wisdom in circular intervals. https://www.amazon.com/Geometry-Biological-Interdisciplinary-Applied-Mathematics/dp/0387989927 https://www.amazon.com/Geometry-Biological-Interdisciplinary... Neural networks learn bad habits from the (-∞,∞) range, particularly like polynomials they like to make big coefficients that make terms that cancel out to get precise answers. I see all the states on an LSTM creep up in magnitude as it scans a document and that is really just wrong. It makes me think that FP16 is a joke, you get some regularization by not letting the numbers get too big.