3 ms·
Show HN: Long context window for LLMs
- K0IN 3y agowhat happens around 512 tokens? how can rope jump this far while fft is at its lowest?
- alagagbar 3y agoIt's where RoPE fails to generalize to longer contexts, because this method introduces additional data - a data required to encode position. FFT method on the other hand does not introduce any additional data - it learns the position from the available data (input) using long convolutions with wide kernels (it generates those kernels from the data). If you scroll down the notebook, you'll see the generation output made by RoPE and generation made by FFT and what happens to the text generated with RoPE for longer and longer sequences.