It dependends on the context (hah). For, say, continuing a story, even a small LLM will benefit from a mega context even if it struggles to retrieve specific i

It dependends on the context (hah). For, say, continuing a story, even a small LLM will benefit from a mega context even if it struggles to retrieve specific information... As long as perplexity isnt hit too much, which depends on the method. But this is more of an issue for summarization or information retrieval.