3 ms·
Doesn't this imply some kind of scaling law? An LLM of complexity Y will only benefit from a context length of at most Z? That feels truthy to me.
by regularfry 3y ago
Doesn't this imply some kind of scaling law? An LLM of complexity Y will only benefit from a context length of at most Z? That feels truthy to me.
- brucethemoose2 3y agoIt dependends on the context (hah). For, say, continuing a story, even a small LLM will benefit from a mega context even if it struggles to retrieve specific information... As long as perplexity isnt hit too much, which depends on the method. But this is more of an issue for summarization or information retrieval.