7 ms·
My understanding is that in NTK aware RoPE scaling, the model does pay uniform attention. With older methods, not as much.
by npsomaratna 3y ago
My understanding is that in NTK aware RoPE scaling, the model does pay uniform attention. With older methods, not as much.