alagagbar

4 ms·

1. 1.▲
   by alagagbar 3y ago

   It's where RoPE fails to generalize to longer contexts, because this method introduces additional data - a data required to encode position. FFT method on the other hand does not introduce any additional data - it learns the position f
2. 2.▲
   Show HN: Long context window for LLMs(colab.research.google.com)

   6 points by alagagbar 3y ago|2 comments