3 ms·
> developed techniques like Long LoRA which make theoretical disadvantages functionally irrelevant. How’s that?
by bilsbie 3y ago
> developed techniques like Long LoRA which make theoretical disadvantages functionally irrelevant.
How’s that?
- janalsncm 3y agoContext windows used to be tiny e.g. 768 or 1024. That meant sliding windows of limited context if you had any reasonably sized input. If your context window is 32k or even 100k, a lot of inputs will fit into the context window entirely. The reason huge context windows weren't possible in the past is that memory requirements were quadratic with input length. Long LoRA lets us use less memory for our context windows, or use the same memory footprint for larger context windows.